Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 13:58:44.496955 398 test_util.cc:268] Using random seed: 506405412
I20240503 13:58:44.504083 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:58:44.504213 398 ts_itest-base.cc:113] --------------
I20240503 13:58:44.504247 398 ts_itest-base.cc:114] 6 tablet servers
I20240503 13:58:44.504266 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:58:44.504278 398 ts_itest-base.cc:116] --------------
2024-05-03T13:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:58:44Z Disabled control of system clock
I20240503 13:58:44.513393 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37145
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37145
--location_mapping_cmd=/tmp/dist-test-taskmJWZN5/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 13:58:44.629657 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.629937 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.630007 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.633333 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.633430 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:58:44.633460 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.633478 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:58:44.633517 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:58:44.637537 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskmJWZN5/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37145
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.638680 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.639928 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.648257 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.648785 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.649344 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.650213 410 server_base.cc:1034] running on GCE node
I20240503 13:58:44.650799 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.651896 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.653044 410 hybrid_clock.cc:648] HybridClock initialized: now 1714744724653008 us; error 58 us; skew 500 ppm
I20240503 13:58:44.660449 410 webserver.cc:460] Webserver started at http://127.0.99.190:46851/ using document root <none> and password file <none>
I20240503 13:58:44.661099 410 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.661164 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.661370 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.663252 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "deece84684b0459cae6e75399d829767"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-6fwv"
I20240503 13:58:44.663789 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "deece84684b0459cae6e75399d829767"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-6fwv"
I20240503 13:58:44.667465 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:44.669891 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.671049 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:44.671185 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "deece84684b0459cae6e75399d829767"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-6fwv"
I20240503 13:58:44.671288 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.685580 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.686404 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.686599 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.693521 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37145 every 8 connection(s)
I20240503 13:58:44.693527 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37145
I20240503 13:58:44.694754 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:58:44.697585 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.699874 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 410
I20240503 13:58:44.700035 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:58:44.704377 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767: Bootstrap starting.
I20240503 13:58:44.707031 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.707907 477 log.cc:824] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:44.710026 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767: No bootstrap required, opened a new log
I20240503 13:58:44.713229 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deece84684b0459cae6e75399d829767" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37145 } }
I20240503 13:58:44.713461 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.713512 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deece84684b0459cae6e75399d829767, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.714071 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deece84684b0459cae6e75399d829767" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37145 } }
I20240503 13:58:44.714254 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:44.714315 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:44.714428 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.715394 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deece84684b0459cae6e75399d829767" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37145 } }
I20240503 13:58:44.715742 477 leader_election.cc:302] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deece84684b0459cae6e75399d829767; no voters:
I20240503 13:58:44.716041 477 leader_election.cc:288] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:44.716123 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.716307 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [term 1 LEADER]: Becoming Leader. State: Replica: deece84684b0459cae6e75399d829767, State: Running, Role: LEADER
I20240503 13:58:44.716671 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deece84684b0459cae6e75399d829767" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37145 } }
I20240503 13:58:44.717329 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:58:44.717922 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deece84684b0459cae6e75399d829767. Latest consensus state: current_term: 1 leader_uuid: "deece84684b0459cae6e75399d829767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deece84684b0459cae6e75399d829767" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37145 } } }
I20240503 13:58:44.718062 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:44.718364 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deece84684b0459cae6e75399d829767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deece84684b0459cae6e75399d829767" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37145 } } }
I20240503 13:58:44.718444 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:58:44.718473 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767 [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:44.721637 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:58:44.727840 486 catalog_manager.cc:1347] Generated new cluster ID: f5a1ad4993a242ac970f6794534904da
I20240503 13:58:44.727919 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:58:44.743032 486 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:58:44.744086 486 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:58:44.758023 486 catalog_manager.cc:5857] T 00000000000000000000000000000000 P deece84684b0459cae6e75399d829767: Generated new TSK 0
I20240503 13:58:44.758845 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:58:44.771304 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:44.898042 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.898305 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:44.898352 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:44.901537 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:44.901615 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:44.901690 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:58:44.905580 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:44.906584 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:44.907783 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:44.916033 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.916622 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:44.917116 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:44.918052 501 server_base.cc:1034] running on GCE node
I20240503 13:58:44.918454 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:44.919071 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:44.920222 501 hybrid_clock.cc:648] HybridClock initialized: now 1714744724920198 us; error 39 us; skew 500 ppm
I20240503 13:58:44.922145 501 webserver.cc:460] Webserver started at http://127.0.99.129:35519/ using document root <none> and password file <none>
I20240503 13:58:44.922720 501 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:44.922776 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:44.922964 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:44.924682 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-6fwv"
I20240503 13:58:44.925161 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-6fwv"
I20240503 13:58:44.928637 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20240503 13:58:44.930845 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.931926 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:58:44.932047 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
format_stamp: "Formatted at 2024-05-03 13:58:44 on dist-test-slave-6fwv"
I20240503 13:58:44.932140 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:44.963554 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:44.964303 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:44.964483 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:44.965085 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:44.966106 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:44.966158 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.966214 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:44.966238 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:44.974947 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37227
I20240503 13:58:44.974995 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37227 every 8 connection(s)
I20240503 13:58:44.975996 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:58:44.978124 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 501
I20240503 13:58:44.978235 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:58:44.980561 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:44.986121 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37145
I20240503 13:58:44.986423 629 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:44.987207 629 heartbeater.cc:511] Master 127.0.99.190:37145 requested a full tablet report, sending...
I20240503 13:58:45.036872 441 ts_manager.cc:194] Registered new tserver with Master: 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:45.038300 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:33535
W20240503 13:58:45.111537 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.111795 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.111843 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.115244 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.115331 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.115414 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:58:45.119508 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.120555 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.121793 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.131016 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.131527 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.132299 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.133049 632 server_base.cc:1034] running on GCE node
I20240503 13:58:45.133558 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.134253 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.135398 632 hybrid_clock.cc:648] HybridClock initialized: now 1714744725135380 us; error 38 us; skew 500 ppm
I20240503 13:58:45.137446 632 webserver.cc:460] Webserver started at http://127.0.99.130:40803/ using document root <none> and password file <none>
I20240503 13:58:45.138101 632 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.138166 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.138360 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.140188 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "02d3337c0dea4194926aa98965f3bebb"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.140713 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "02d3337c0dea4194926aa98965f3bebb"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.144429 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:45.147024 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.148221 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 13:58:45.148358 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "02d3337c0dea4194926aa98965f3bebb"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.148468 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.182953 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.183708 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.183887 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.184468 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.185461 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.185513 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.185577 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.185601 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.195456 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38551
I20240503 13:58:45.195490 760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38551 every 8 connection(s)
I20240503 13:58:45.196484 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:58:45.196774 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 632
I20240503 13:58:45.196877 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:58:45.199204 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:45.207803 761 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37145
I20240503 13:58:45.208128 761 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.208902 761 heartbeater.cc:511] Master 127.0.99.190:37145 requested a full tablet report, sending...
I20240503 13:58:45.247725 441 ts_manager.cc:194] Registered new tserver with Master: 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:45.248793 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:40581
W20240503 13:58:45.319540 764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.319790 764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.319837 764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.323061 764 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.323145 764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.323216 764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:45.327433 764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--never_fsync=true
--heap_profile_path=/tmp/kudu.764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.328433 764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.329640 764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.338724 773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.339169 771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.339920 770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.340493 764 server_base.cc:1034] running on GCE node
I20240503 13:58:45.340941 764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.341588 764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.342737 764 hybrid_clock.cc:648] HybridClock initialized: now 1714744725342721 us; error 34 us; skew 500 ppm
I20240503 13:58:45.344699 764 webserver.cc:460] Webserver started at http://127.0.99.131:33753/ using document root <none> and password file <none>
I20240503 13:58:45.345292 764 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.345351 764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.345541 764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.347411 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0975c6ac192d4f90bf114279c55bd856"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.347904 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0975c6ac192d4f90bf114279c55bd856"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.351404 764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:45.353879 779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.355047 764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:45.355165 764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0975c6ac192d4f90bf114279c55bd856"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.355263 764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.391110 764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.391837 764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.392011 764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.392588 764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.393548 764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.393599 764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.393653 764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.393682 764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.403951 764 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34451
I20240503 13:58:45.404006 892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34451 every 8 connection(s)
I20240503 13:58:45.404971 764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:45.413791 893 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37145
I20240503 13:58:45.414075 893 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.414664 893 heartbeater.cc:511] Master 127.0.99.190:37145 requested a full tablet report, sending...
I20240503 13:58:45.415256 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 764
I20240503 13:58:45.415366 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:58:45.419744 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:45.450064 441 ts_manager.cc:194] Registered new tserver with Master: 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:45.450771 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49515
W20240503 13:58:45.533905 897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.534266 897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.534327 897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.537639 897 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.537720 897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.537806 897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:58:45.541718 897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--never_fsync=true
--heap_profile_path=/tmp/kudu.897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.542732 897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.543889 897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.552639 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.553258 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.554090 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.554592 897 server_base.cc:1034] running on GCE node
I20240503 13:58:45.555032 897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.555656 897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.556823 897 hybrid_clock.cc:648] HybridClock initialized: now 1714744725556792 us; error 46 us; skew 500 ppm
I20240503 13:58:45.558841 897 webserver.cc:460] Webserver started at http://127.0.99.132:42369/ using document root <none> and password file <none>
I20240503 13:58:45.559432 897 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.559500 897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.559710 897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.561494 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0f2cb90442f04ba0944c9fbc78943c21"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.562031 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0f2cb90442f04ba0944c9fbc78943c21"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.565608 897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 13:58:45.568095 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.569321 897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:45.569447 897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0f2cb90442f04ba0944c9fbc78943c21"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.569556 897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.582203 897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.582968 897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.583144 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.583752 897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.584748 897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.584818 897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.584887 897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.584910 897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.594468 897 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:42789
I20240503 13:58:45.594511 1024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:42789 every 8 connection(s)
I20240503 13:58:45.595539 897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:58:45.605160 1025 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37145
I20240503 13:58:45.605324 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 897
I20240503 13:58:45.605430 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:58:45.605455 1025 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.606200 1025 heartbeater.cc:511] Master 127.0.99.190:37145 requested a full tablet report, sending...
I20240503 13:58:45.607648 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:45.641505 441 ts_manager.cc:194] Registered new tserver with Master: 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:45.642508 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:56499
W20240503 13:58:45.731871 1029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.732115 1029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.732160 1029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.735483 1029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.735574 1029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.735648 1029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:45.739720 1029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--never_fsync=true
--heap_profile_path=/tmp/kudu.1029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.740777 1029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.742039 1029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.751111 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.751410 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.752213 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.752945 1029 server_base.cc:1034] running on GCE node
I20240503 13:58:45.753449 1029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.754135 1029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.755303 1029 hybrid_clock.cc:648] HybridClock initialized: now 1714744725755277 us; error 43 us; skew 500 ppm
I20240503 13:58:45.757314 1029 webserver.cc:460] Webserver started at http://127.0.99.133:43957/ using document root <none> and password file <none>
I20240503 13:58:45.757915 1029 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.757995 1029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.758211 1029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.760042 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ced9864d089c400e9bfaf3d07bb70acd"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.760537 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ced9864d089c400e9bfaf3d07bb70acd"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.764115 1029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:45.766729 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.767937 1029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:45.768076 1029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ced9864d089c400e9bfaf3d07bb70acd"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.768289 1029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.783507 1029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.784260 1029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.784435 1029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.785012 1029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.786011 1029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.786069 1029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.786128 1029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.786152 1029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.796536 1029 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:44095
I20240503 13:58:45.796571 1156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:44095 every 8 connection(s)
I20240503 13:58:45.797564 1029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:45.806819 1157 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37145
I20240503 13:58:45.807063 1157 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.807715 1157 heartbeater.cc:511] Master 127.0.99.190:37145 requested a full tablet report, sending...
I20240503 13:58:45.807883 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 1029
I20240503 13:58:45.807989 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:58:45.812377 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--builtin_ntp_servers=127.0.99.148:33719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:45.843776 441 ts_manager.cc:194] Registered new tserver with Master: ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:45.844743 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:33023
W20240503 13:58:45.928988 1161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.929229 1161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:45.929272 1161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.932498 1161 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:45.932585 1161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:45.932658 1161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:58:45.936553 1161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37145
--never_fsync=true
--heap_profile_path=/tmp/kudu.1161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:45.937525 1161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:45.938730 1161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:45.947242 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.947926 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:45.948446 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:45.949004 1161 server_base.cc:1034] running on GCE node
I20240503 13:58:45.949409 1161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:45.950073 1161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:45.951231 1161 hybrid_clock.cc:648] HybridClock initialized: now 1714744725951213 us; error 33 us; skew 500 ppm
I20240503 13:58:45.953164 1161 webserver.cc:460] Webserver started at http://127.0.99.134:45655/ using document root <none> and password file <none>
I20240503 13:58:45.953747 1161 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:45.953807 1161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:45.954025 1161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:45.956331 1161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "7d435873bcf04b3d9901363d8192342b"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.956914 1161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "7d435873bcf04b3d9901363d8192342b"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.960655 1161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20240503 13:58:45.962961 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.964020 1161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240503 13:58:45.964138 1161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "7d435873bcf04b3d9901363d8192342b"
format_stamp: "Formatted at 2024-05-03 13:58:45 on dist-test-slave-6fwv"
I20240503 13:58:45.964226 1161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:45.975893 1161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:45.976565 1161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:45.976722 1161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:45.977252 1161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:45.978199 1161 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:45.978250 1161 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.978302 1161 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:45.978328 1161 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:45.987421 1161 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40511
I20240503 13:58:45.987444 1288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40511 every 8 connection(s)
I20240503 13:58:45.988435 1161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:58:45.998154 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 1161
I20240503 13:58:45.998277 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:58:45.998536 1289 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37145
I20240503 13:58:45.998780 1289 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:45.999483 1289 heartbeater.cc:511] Master 127.0.99.190:37145 requested a full tablet report, sending...
I20240503 13:58:46.034860 440 ts_manager.cc:194] Registered new tserver with Master: 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:46.035670 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39747
I20240503 13:58:46.037290 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 13:58:46.041267 629 heartbeater.cc:503] Master 127.0.99.190:37145 was elected leader, sending a full tablet report...
I20240503 13:58:46.049535 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 13:58:46.251551 761 heartbeater.cc:503] Master 127.0.99.190:37145 was elected leader, sending a full tablet report...
I20240503 13:58:46.453549 893 heartbeater.cc:503] Master 127.0.99.190:37145 was elected leader, sending a full tablet report...
I20240503 13:58:46.644893 1025 heartbeater.cc:503] Master 127.0.99.190:37145 was elected leader, sending a full tablet report...
I20240503 13:58:46.847307 1157 heartbeater.cc:503] Master 127.0.99.190:37145 was elected leader, sending a full tablet report...
I20240503 13:58:47.038520 1289 heartbeater.cc:503] Master 127.0.99.190:37145 was elected leader, sending a full tablet report...
I20240503 13:58:49.802318 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:49.825171 562 tablet_service.cc:1467] Processing CreateTablet for tablet cadab85bb9e84c4bb2c162e7d262c681 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.825170 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 880ae94bf80a4eaa932ca387c8a7bb51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.826017 563 tablet_service.cc:1467] Processing CreateTablet for tablet 880ae94bf80a4eaa932ca387c8a7bb51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.826323 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadab85bb9e84c4bb2c162e7d262c681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.826678 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880ae94bf80a4eaa932ca387c8a7bb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.828332 1091 tablet_service.cc:1467] Processing CreateTablet for tablet fe2a2afa277d4ebfa3069396de51bbb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.829309 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2a2afa277d4ebfa3069396de51bbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.830367 1090 tablet_service.cc:1467] Processing CreateTablet for tablet cadab85bb9e84c4bb2c162e7d262c681 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.830648 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadab85bb9e84c4bb2c162e7d262c681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.832103 959 tablet_service.cc:1467] Processing CreateTablet for tablet 880ae94bf80a4eaa932ca387c8a7bb51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.833106 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880ae94bf80a4eaa932ca387c8a7bb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.833293 827 tablet_service.cc:1467] Processing CreateTablet for tablet fe2a2afa277d4ebfa3069396de51bbb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.834221 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2a2afa277d4ebfa3069396de51bbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.834321 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880ae94bf80a4eaa932ca387c8a7bb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.835717 826 tablet_service.cc:1467] Processing CreateTablet for tablet cadab85bb9e84c4bb2c162e7d262c681 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.835969 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadab85bb9e84c4bb2c162e7d262c681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.838892 1314 tablet_bootstrap.cc:492] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:49.841553 1314 tablet_bootstrap.cc:654] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.842720 1314 log.cc:824] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.844342 695 tablet_service.cc:1467] Processing CreateTablet for tablet fe2a2afa277d4ebfa3069396de51bbb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=98a1ac4f1d9b451c95cab7e5d59a323e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:49.845144 1315 tablet_bootstrap.cc:492] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:49.845387 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2a2afa277d4ebfa3069396de51bbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.846256 1318 tablet_bootstrap.cc:492] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:49.848611 1318 tablet_bootstrap.cc:654] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.849128 1314 tablet_bootstrap.cc:492] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:49.849182 1315 tablet_bootstrap.cc:654] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.849429 1314 ts_tablet_manager.cc:1367] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 13:58:49.849448 1318 log.cc:824] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.850059 1315 log.cc:824] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.850265 1319 tablet_bootstrap.cc:492] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:49.850682 1316 tablet_bootstrap.cc:492] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:49.852223 1315 tablet_bootstrap.cc:492] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:49.852418 1315 ts_tablet_manager.cc:1367] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 13:58:49.854337 1322 tablet_bootstrap.cc:492] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:49.854493 1318 tablet_bootstrap.cc:492] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:49.854701 1318 ts_tablet_manager.cc:1367] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 13:58:49.855403 1314 raft_consensus.cc:348] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.855806 1314 raft_consensus.cc:374] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.855916 1314 raft_consensus.cc:729] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.856318 1315 raft_consensus.cc:348] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.856598 1315 raft_consensus.cc:374] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.856688 1315 raft_consensus.cc:729] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.856540 1314 consensus_queue.cc:260] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.856920 1322 tablet_bootstrap.cc:654] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.857146 1315 consensus_queue.cc:260] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.857822 1314 ts_tablet_manager.cc:1398] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 13:58:49.857878 1322 log.cc:824] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.858093 1315 ts_tablet_manager.cc:1398] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240503 13:58:49.858489 1315 tablet_bootstrap.cc:492] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:49.859583 1315 tablet_bootstrap.cc:654] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.861079 1322 tablet_bootstrap.cc:492] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:49.861330 1322 ts_tablet_manager.cc:1367] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 13:58:49.861584 1319 tablet_bootstrap.cc:654] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.861584 1316 tablet_bootstrap.cc:654] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.862529 1319 log.cc:824] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.862529 1316 log.cc:824] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:49.862591 1314 tablet_bootstrap.cc:492] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:49.863482 1318 raft_consensus.cc:348] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.863799 1318 raft_consensus.cc:374] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.863896 1318 raft_consensus.cc:729] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.864074 1315 tablet_bootstrap.cc:492] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:49.864204 1315 ts_tablet_manager.cc:1367] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:49.864441 1318 consensus_queue.cc:260] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.864842 1315 raft_consensus.cc:348] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.864985 1315 raft_consensus.cc:374] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.865051 1315 raft_consensus.cc:729] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.865199 1315 consensus_queue.cc:260] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.865492 1318 ts_tablet_manager.cc:1398] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240503 13:58:49.865537 1315 ts_tablet_manager.cc:1398] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:49.865903 1318 tablet_bootstrap.cc:492] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:49.866838 1314 tablet_bootstrap.cc:654] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.867617 1318 tablet_bootstrap.cc:654] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:49.867954 1322 raft_consensus.cc:348] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.868275 1322 raft_consensus.cc:374] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.868374 1322 raft_consensus.cc:729] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.868983 1322 consensus_queue.cc:260] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.869349 1319 tablet_bootstrap.cc:492] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:49.869550 1316 tablet_bootstrap.cc:492] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:49.869566 1319 ts_tablet_manager.cc:1367] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240503 13:58:49.869699 1316 ts_tablet_manager.cc:1367] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 13:58:49.871937 1324 raft_consensus.cc:482] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:49.872110 1324 raft_consensus.cc:504] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.873687 1319 raft_consensus.cc:348] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.873687 1316 raft_consensus.cc:348] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.873926 1319 raft_consensus.cc:374] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.873925 1316 raft_consensus.cc:374] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.874004 1316 raft_consensus.cc:729] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.874004 1319 raft_consensus.cc:729] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.874616 1316 consensus_queue.cc:260] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.875093 1319 consensus_queue.cc:260] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.875687 1316 ts_tablet_manager.cc:1398] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:58:49.876042 1319 ts_tablet_manager.cc:1398] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:58:49.876534 1324 leader_election.cc:288] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:49.878410 1318 tablet_bootstrap.cc:492] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:49.878409 1314 tablet_bootstrap.cc:492] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:49.878513 1318 ts_tablet_manager.cc:1367] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20240503 13:58:49.878513 1314 ts_tablet_manager.cc:1367] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:58:49.878640 1322 ts_tablet_manager.cc:1398] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.017s user 0.000s sys 0.007s
I20240503 13:58:49.879118 1318 raft_consensus.cc:348] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.879118 1314 raft_consensus.cc:348] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.879230 1318 raft_consensus.cc:374] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.879230 1314 raft_consensus.cc:374] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:49.879259 1314 raft_consensus.cc:729] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.879259 1318 raft_consensus.cc:729] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:49.879384 1318 consensus_queue.cc:260] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.879384 1314 consensus_queue.cc:260] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.879637 1314 ts_tablet_manager.cc:1398] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:49.879654 1318 ts_tablet_manager.cc:1398] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:49.883816 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadab85bb9e84c4bb2c162e7d262c681" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:49.884207 583 raft_consensus.cc:2454] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:49.884732 1047 leader_election.cc:302] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:49.884917 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadab85bb9e84c4bb2c162e7d262c681" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:49.885044 1324 raft_consensus.cc:2781] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:49.885143 1324 raft_consensus.cc:482] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:49.885195 1324 raft_consensus.cc:3037] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.885172 847 raft_consensus.cc:2454] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
W20240503 13:58:49.909195 894 tablet.cc:2369] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:49.924177 1331 raft_consensus.cc:482] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:49.924386 1331 raft_consensus.cc:504] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.925572 1331 leader_election.cc:288] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:49.928853 1323 raft_consensus.cc:482] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:49.929011 1323 raft_consensus.cc:504] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.929400 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a2afa277d4ebfa3069396de51bbb7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:49.929546 847 raft_consensus.cc:2454] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:49.929512 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a2afa277d4ebfa3069396de51bbb7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:49.929723 1111 raft_consensus.cc:2454] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:49.929936 651 leader_election.cc:302] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0975c6ac192d4f90bf114279c55bd856; no voters:
I20240503 13:58:49.930275 1331 raft_consensus.cc:2781] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:49.930361 1331 raft_consensus.cc:482] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:49.930408 1331 raft_consensus.cc:3037] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.930485 1323 leader_election.cc:288] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:49.934728 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ae94bf80a4eaa932ca387c8a7bb51" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:49.935111 1243 raft_consensus.cc:2454] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:49.935266 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ae94bf80a4eaa932ca387c8a7bb51" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:49.935555 979 raft_consensus.cc:2454] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:49.935632 517 leader_election.cc:302] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:49.935922 1323 raft_consensus.cc:2781] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:49.936008 1323 raft_consensus.cc:482] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:49.936056 1323 raft_consensus.cc:3037] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:58:49.951251 762 tablet.cc:2369] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:49.966325 1331 raft_consensus.cc:504] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.966325 1323 raft_consensus.cc:504] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:49.966325 1324 raft_consensus.cc:504] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.966862 1323 leader_election.cc:288] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:49.966868 1331 leader_election.cc:288] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:49.967551 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a2afa277d4ebfa3069396de51bbb7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:49.967710 847 raft_consensus.cc:3037] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.967911 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadab85bb9e84c4bb2c162e7d262c681" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:49.968235 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadab85bb9e84c4bb2c162e7d262c681" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:49.968396 583 raft_consensus.cc:3037] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.969177 847 raft_consensus.cc:2454] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:49.969679 651 leader_election.cc:302] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0975c6ac192d4f90bf114279c55bd856; no voters:
I20240503 13:58:49.970003 1331 raft_consensus.cc:2781] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.969789 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ae94bf80a4eaa932ca387c8a7bb51" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:49.970245 1331 raft_consensus.cc:686] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:49.970278 1243 raft_consensus.cc:3037] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.970661 1331 consensus_queue.cc:237] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:49.971246 583 raft_consensus.cc:2454] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:49.968021 846 raft_consensus.cc:3037] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.966851 1324 leader_election.cc:288] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:49.967447 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a2afa277d4ebfa3069396de51bbb7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:49.972442 1111 raft_consensus.cc:3037] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.971727 1047 leader_election.cc:302] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:49.972678 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880ae94bf80a4eaa932ca387c8a7bb51" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:49.972832 979 raft_consensus.cc:3037] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:49.973021 1324 raft_consensus.cc:2781] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:49.973269 1324 raft_consensus.cc:686] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [term 1 LEADER]: Becoming Leader. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Running, Role: LEADER
I20240503 13:58:49.973877 1324 consensus_queue.cc:237] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:49.975401 439 catalog_manager.cc:5484] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:49.977444 441 catalog_manager.cc:5484] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd reported cstate change: term changed from 0 to 1, leader changed from <none> to ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "ced9864d089c400e9bfaf3d07bb70acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:49.980952 630 tablet.cc:2369] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:58:49.992128 1290 tablet.cc:2369] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:58:50.051661 1158 tablet.cc:2369] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:50.063551 846 raft_consensus.cc:2454] T cadab85bb9e84c4bb2c162e7d262c681 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:50.063551 1243 raft_consensus.cc:2454] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:50.063551 1111 raft_consensus.cc:2454] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:50.064051 517 leader_election.cc:302] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:50.064301 1323 raft_consensus.cc:2781] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.064318 979 raft_consensus.cc:2454] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:50.064563 1323 raft_consensus.cc:686] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:50.064970 1323 consensus_queue.cc:237] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.068625 441 catalog_manager.cc:5484] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:50.100486 1026 tablet.cc:2369] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:58:50.244937 1347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.245229 1347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.249450 1347 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 13:58:50.346210 1111 raft_consensus.cc:1261] T fe2a2afa277d4ebfa3069396de51bbb7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.346333 846 raft_consensus.cc:1261] T fe2a2afa277d4ebfa3069396de51bbb7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.347108 1331 consensus_queue.cc:1035] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.347832 1331 consensus_queue.cc:1035] T fe2a2afa277d4ebfa3069396de51bbb7 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.349967 979 raft_consensus.cc:1261] T 880ae94bf80a4eaa932ca387c8a7bb51 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.350675 1243 raft_consensus.cc:1261] T 880ae94bf80a4eaa932ca387c8a7bb51 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.351187 1323 consensus_queue.cc:1035] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.352141 1324 consensus_queue.cc:1035] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.356719 1378 mvcc.cc:205] Tried to move back new op lower bound from 7023594415488823296 to 7023594413964623872. Current Snapshot: MvccSnapshot[applied={T|T < 7023594415488823296}]
I20240503 13:58:50.358111 1323 consensus_queue.cc:1035] T 880ae94bf80a4eaa932ca387c8a7bb51 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.359076 1381 mvcc.cc:205] Tried to move back new op lower bound from 7023594415488823296 to 7023594413964623872. Current Snapshot: MvccSnapshot[applied={T|T < 7023594415488823296}]
I20240503 13:58:50.361265 583 raft_consensus.cc:1261] T cadab85bb9e84c4bb2c162e7d262c681 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.362499 1324 consensus_queue.cc:1035] T cadab85bb9e84c4bb2c162e7d262c681 P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.399103 1390 mvcc.cc:205] Tried to move back new op lower bound from 7023594415501361152 to 7023594414349062144. Current Snapshot: MvccSnapshot[applied={T|T < 7023594415501361152}]
I20240503 13:58:50.401751 1387 mvcc.cc:205] Tried to move back new op lower bound from 7023594415501361152 to 7023594414349062144. Current Snapshot: MvccSnapshot[applied={T|T < 7023594415501361152}]
I20240503 13:58:50.407603 1394 mvcc.cc:205] Tried to move back new op lower bound from 7023594415612960768 to 7023594414349062144. Current Snapshot: MvccSnapshot[applied={T|T < 7023594415612960768 or (T in {7023594415612960768})}]
INSERT report
rows total: 2000
time total: 137.786 ms
time per row: 0.0688928 ms
I20240503 13:58:50.440188 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:50.446491 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 1188c7f43c854198b189b0110cc57f93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.446491 959 tablet_service.cc:1467] Processing CreateTablet for tablet 1188c7f43c854198b189b0110cc57f93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.446835 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1188c7f43c854198b189b0110cc57f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.446835 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1188c7f43c854198b189b0110cc57f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.447257 1091 tablet_service.cc:1467] Processing CreateTablet for tablet c6f32248db054f8d80934d6e6fb22454 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.447434 827 tablet_service.cc:1467] Processing CreateTablet for tablet c6f32248db054f8d80934d6e6fb22454 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.447551 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f32248db054f8d80934d6e6fb22454. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.447677 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f32248db054f8d80934d6e6fb22454. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.449752 1411 tablet_bootstrap.cc:492] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:50.449934 1410 tablet_bootstrap.cc:492] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:50.450747 1411 tablet_bootstrap.cc:654] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.450670 562 tablet_service.cc:1467] Processing CreateTablet for tablet c6f32248db054f8d80934d6e6fb22454 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.450872 1410 tablet_bootstrap.cc:654] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.450984 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f32248db054f8d80934d6e6fb22454. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.451598 1409 tablet_bootstrap.cc:492] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:50.451747 1411 tablet_bootstrap.cc:492] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:50.451836 1411 ts_tablet_manager.cc:1367] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.452354 1411 raft_consensus.cc:348] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.452458 1411 raft_consensus.cc:374] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.452488 1411 raft_consensus.cc:729] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.452652 1411 consensus_queue.cc:260] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.452891 1411 ts_tablet_manager.cc:1398] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:50.453588 1415 tablet_bootstrap.cc:492] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:50.453814 1410 tablet_bootstrap.cc:492] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:50.453841 959 tablet_service.cc:1467] Processing CreateTablet for tablet 746c4677ebaf4cb4bd9bd69682d423a0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.453923 1410 ts_tablet_manager.cc:1367] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:58:50.454154 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746c4677ebaf4cb4bd9bd69682d423a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.454492 1415 tablet_bootstrap.cc:654] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.454569 1410 raft_consensus.cc:348] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.454694 1410 raft_consensus.cc:374] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.454726 1410 raft_consensus.cc:729] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.454851 1410 consensus_queue.cc:260] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.455114 1410 ts_tablet_manager.cc:1398] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:50.455302 1409 tablet_bootstrap.cc:654] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.455828 1415 tablet_bootstrap.cc:492] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:50.455929 1415 ts_tablet_manager.cc:1367] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.456354 1415 raft_consensus.cc:348] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.456460 1415 raft_consensus.cc:374] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.456490 1415 raft_consensus.cc:729] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.456511 1409 tablet_bootstrap.cc:492] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:50.456636 1409 ts_tablet_manager.cc:1367] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:50.456619 1415 consensus_queue.cc:260] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.456861 1415 ts_tablet_manager.cc:1398] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:50.457289 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 746c4677ebaf4cb4bd9bd69682d423a0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.457559 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746c4677ebaf4cb4bd9bd69682d423a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.457587 695 tablet_service.cc:1467] Processing CreateTablet for tablet 1188c7f43c854198b189b0110cc57f93 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.457757 1409 raft_consensus.cc:348] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.457856 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1188c7f43c854198b189b0110cc57f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.457855 1409 raft_consensus.cc:374] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.457886 1409 raft_consensus.cc:729] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.458065 1409 consensus_queue.cc:260] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.458317 1409 ts_tablet_manager.cc:1398] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.458467 1409 tablet_bootstrap.cc:492] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:50.459501 1409 tablet_bootstrap.cc:654] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.459755 1410 tablet_bootstrap.cc:492] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:50.460325 694 tablet_service.cc:1467] Processing CreateTablet for tablet 746c4677ebaf4cb4bd9bd69682d423a0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2385cb9fe6f545f988c100b3eb5d72ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:50.460667 1410 tablet_bootstrap.cc:654] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.460734 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746c4677ebaf4cb4bd9bd69682d423a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:50.462157 1410 tablet_bootstrap.cc:492] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:50.462249 1410 ts_tablet_manager.cc:1367] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:58:50.463585 1410 raft_consensus.cc:348] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.463708 1410 raft_consensus.cc:374] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.463734 1410 raft_consensus.cc:729] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.463876 1410 consensus_queue.cc:260] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.463912 1418 tablet_bootstrap.cc:492] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:50.464179 1410 ts_tablet_manager.cc:1398] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:50.464686 1409 tablet_bootstrap.cc:492] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:50.464781 1409 ts_tablet_manager.cc:1367] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:50.464888 1418 tablet_bootstrap.cc:654] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.465660 1412 tablet_bootstrap.cc:492] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:50.466763 1412 tablet_bootstrap.cc:654] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.467160 1409 raft_consensus.cc:348] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.467329 1409 raft_consensus.cc:374] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.467365 1409 raft_consensus.cc:729] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.467525 1409 consensus_queue.cc:260] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.468228 1409 ts_tablet_manager.cc:1398] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:50.468573 1412 tablet_bootstrap.cc:492] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:50.468683 1412 ts_tablet_manager.cc:1367] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:50.469079 1418 tablet_bootstrap.cc:492] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:50.469172 1418 ts_tablet_manager.cc:1367] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:50.469172 1412 raft_consensus.cc:348] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.469285 1412 raft_consensus.cc:374] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.469323 1412 raft_consensus.cc:729] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.469455 1412 consensus_queue.cc:260] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.469820 1418 raft_consensus.cc:348] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.469924 1418 raft_consensus.cc:374] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.469954 1418 raft_consensus.cc:729] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.470106 1418 consensus_queue.cc:260] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.470346 1412 ts_tablet_manager.cc:1398] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.470815 1418 ts_tablet_manager.cc:1398] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.471027 1418 tablet_bootstrap.cc:492] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:50.472072 1418 tablet_bootstrap.cc:654] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:50.473269 1418 tablet_bootstrap.cc:492] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:50.473361 1418 ts_tablet_manager.cc:1367] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:50.473804 1418 raft_consensus.cc:348] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.473896 1418 raft_consensus.cc:374] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:50.473927 1418 raft_consensus.cc:729] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:50.474072 1418 consensus_queue.cc:260] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.474323 1418 ts_tablet_manager.cc:1398] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:50.490409 1408 raft_consensus.cc:482] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:50.490546 1408 raft_consensus.cc:504] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.491060 1408 leader_election.cc:288] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:50.494850 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746c4677ebaf4cb4bd9bd69682d423a0" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:50.495046 1243 raft_consensus.cc:2454] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:50.495209 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746c4677ebaf4cb4bd9bd69682d423a0" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:50.495378 979 raft_consensus.cc:2454] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:50.495354 649 leader_election.cc:302] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:50.495558 1408 raft_consensus.cc:2781] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:50.495642 1408 raft_consensus.cc:482] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:50.495693 1408 raft_consensus.cc:3037] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.556288 1331 raft_consensus.cc:482] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:50.556448 1331 raft_consensus.cc:504] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.556910 1331 leader_election.cc:288] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:50.557454 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188c7f43c854198b189b0110cc57f93" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:50.557469 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188c7f43c854198b189b0110cc57f93" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:50.557633 979 raft_consensus.cc:2454] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:50.557627 1243 raft_consensus.cc:2454] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:50.557940 649 leader_election.cc:302] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:50.558198 1331 raft_consensus.cc:2781] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:50.558261 1331 raft_consensus.cc:482] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:50.558288 1331 raft_consensus.cc:3037] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.562410 1408 raft_consensus.cc:504] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.562929 1408 leader_election.cc:288] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:50.563410 1331 raft_consensus.cc:504] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.563513 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746c4677ebaf4cb4bd9bd69682d423a0" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:50.563640 1243 raft_consensus.cc:3037] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.563896 1331 leader_election.cc:288] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:50.563977 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746c4677ebaf4cb4bd9bd69682d423a0" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:50.564117 979 raft_consensus.cc:3037] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.564358 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188c7f43c854198b189b0110cc57f93" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:50.564558 978 raft_consensus.cc:3037] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.564785 1243 raft_consensus.cc:2454] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:50.565059 979 raft_consensus.cc:2454] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:50.565289 649 leader_election.cc:302] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:50.565511 978 raft_consensus.cc:2454] T 1188c7f43c854198b189b0110cc57f93 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:50.565717 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188c7f43c854198b189b0110cc57f93" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:50.565790 1331 raft_consensus.cc:2781] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.565845 1243 raft_consensus.cc:3037] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.565888 1331 raft_consensus.cc:686] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:50.566076 1331 consensus_queue.cc:237] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:50.566442 651 leader_election.cc:302] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:50.566602 1408 raft_consensus.cc:2781] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.566680 1408 raft_consensus.cc:686] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:50.566802 1408 consensus_queue.cc:237] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:50.568182 437 catalog_manager.cc:5484] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:50.574132 437 catalog_manager.cc:5484] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:50.630944 1243 raft_consensus.cc:2454] T 1188c7f43c854198b189b0110cc57f93 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:50.660385 1404 raft_consensus.cc:482] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:50.660542 1404 raft_consensus.cc:504] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.661105 1404 leader_election.cc:288] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:50.664968 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f32248db054f8d80934d6e6fb22454" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:50.664963 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f32248db054f8d80934d6e6fb22454" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:50.665133 1111 raft_consensus.cc:2454] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:50.665138 846 raft_consensus.cc:2454] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:50.665448 518 leader_election.cc:302] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:50.665695 1404 raft_consensus.cc:2781] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:50.665745 1404 raft_consensus.cc:482] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:50.665771 1404 raft_consensus.cc:3037] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.709556 1404 raft_consensus.cc:504] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.710119 1404 leader_election.cc:288] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:50.710664 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f32248db054f8d80934d6e6fb22454" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:50.710691 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f32248db054f8d80934d6e6fb22454" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:50.710803 1111 raft_consensus.cc:3037] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.710841 846 raft_consensus.cc:3037] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:50.711683 1111 raft_consensus.cc:2454] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:50.711889 846 raft_consensus.cc:2454] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:50.712074 518 leader_election.cc:302] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:50.712281 1404 raft_consensus.cc:2781] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:50.712374 1404 raft_consensus.cc:686] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:50.712525 1404 consensus_queue.cc:237] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:50.714497 437 catalog_manager.cc:5484] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:50.871977 1434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:50.872299 1434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:50.875787 1434 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:58:50.946964 1408 consensus_queue.cc:1035] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.952142 1331 consensus_queue.cc:1035] T 1188c7f43c854198b189b0110cc57f93 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:50.991811 978 raft_consensus.cc:1261] T 746c4677ebaf4cb4bd9bd69682d423a0 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.994573 1242 raft_consensus.cc:1261] T 746c4677ebaf4cb4bd9bd69682d423a0 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:50.992913 1408 consensus_queue.cc:1035] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240503 13:58:50.998035 1341 consensus_queue.cc:1035] T 746c4677ebaf4cb4bd9bd69682d423a0 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.034999 846 raft_consensus.cc:1261] T c6f32248db054f8d80934d6e6fb22454 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.035629 1323 consensus_queue.cc:1035] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.037691 1111 raft_consensus.cc:1261] T c6f32248db054f8d80934d6e6fb22454 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.038172 1323 consensus_queue.cc:1035] T c6f32248db054f8d80934d6e6fb22454 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.046238 1485 mvcc.cc:205] Tried to move back new op lower bound from 7023594418279731200 to 7023594416999952384. Current Snapshot: MvccSnapshot[applied={T|T < 7023594418279731200}]
INSERT report
rows total: 2000
time total: 147.556 ms
time per row: 0.0737778 ms
I20240503 13:58:51.071767 435 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:51.078883 827 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab75aa7826a45f6aa3de343a399c02b (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.078893 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab75aa7826a45f6aa3de343a399c02b (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.079267 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab75aa7826a45f6aa3de343a399c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.079267 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab75aa7826a45f6aa3de343a399c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.079845 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a79b904534867bf466729be9a52a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.080152 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a79b904534867bf466729be9a52a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.081634 1497 tablet_bootstrap.cc:492] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:51.081871 1498 tablet_bootstrap.cc:492] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:51.082264 562 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab75aa7826a45f6aa3de343a399c02b (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.082548 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab75aa7826a45f6aa3de343a399c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.082732 1091 tablet_service.cc:1467] Processing CreateTablet for tablet d625f94b7e5e44d1b10b6ad5f1ae9672 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.082918 1496 tablet_bootstrap.cc:492] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:51.082957 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d625f94b7e5e44d1b10b6ad5f1ae9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.083178 694 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a79b904534867bf466729be9a52a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.083460 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a79b904534867bf466729be9a52a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.083912 1496 tablet_bootstrap.cc:654] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.085181 1496 tablet_bootstrap.cc:492] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:51.085268 1496 ts_tablet_manager.cc:1367] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:51.085266 1498 tablet_bootstrap.cc:654] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.085745 1496 raft_consensus.cc:348] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.085814 1501 tablet_bootstrap.cc:492] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:51.085865 1496 raft_consensus.cc:374] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.085893 1496 raft_consensus.cc:729] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.086048 1496 consensus_queue.cc:260] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.086138 1500 tablet_bootstrap.cc:492] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:51.086321 1496 ts_tablet_manager.cc:1398] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.086485 1496 tablet_bootstrap.cc:492] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:51.086789 1501 tablet_bootstrap.cc:654] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.086730 959 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8a79b904534867bf466729be9a52a6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.086937 562 tablet_service.cc:1467] Processing CreateTablet for tablet d625f94b7e5e44d1b10b6ad5f1ae9672 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.087010 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8a79b904534867bf466729be9a52a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.087224 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d625f94b7e5e44d1b10b6ad5f1ae9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.087342 1496 tablet_bootstrap.cc:654] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.088009 1501 tablet_bootstrap.cc:492] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:51.088096 1501 ts_tablet_manager.cc:1367] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:51.088572 1501 raft_consensus.cc:348] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.088678 1501 raft_consensus.cc:374] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.088708 1501 raft_consensus.cc:729] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.088825 1501 consensus_queue.cc:260] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.089079 1501 ts_tablet_manager.cc:1398] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.090262 1500 tablet_bootstrap.cc:654] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.090242 827 tablet_service.cc:1467] Processing CreateTablet for tablet d625f94b7e5e44d1b10b6ad5f1ae9672 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68628afb27de4bc88516983e1f172160]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.090555 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d625f94b7e5e44d1b10b6ad5f1ae9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.091287 1496 tablet_bootstrap.cc:492] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:51.091383 1496 ts_tablet_manager.cc:1367] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:58:51.091792 1496 raft_consensus.cc:348] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.091910 1496 raft_consensus.cc:374] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.091943 1496 raft_consensus.cc:729] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.092063 1496 consensus_queue.cc:260] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.092360 1497 tablet_bootstrap.cc:654] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.092892 1496 ts_tablet_manager.cc:1398] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.093700 1500 tablet_bootstrap.cc:492] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:51.093849 1500 ts_tablet_manager.cc:1367] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:58:51.094365 1500 raft_consensus.cc:348] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.094540 1500 raft_consensus.cc:374] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.094620 1500 raft_consensus.cc:729] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.094950 1505 tablet_bootstrap.cc:492] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:51.095182 1498 tablet_bootstrap.cc:492] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:51.095291 1498 ts_tablet_manager.cc:1367] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:58:51.095701 1498 raft_consensus.cc:348] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.095800 1498 raft_consensus.cc:374] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.095824 1498 raft_consensus.cc:729] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.095873 1505 tablet_bootstrap.cc:654] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.095963 1498 consensus_queue.cc:260] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.096235 1498 ts_tablet_manager.cc:1398] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.096738 1500 consensus_queue.cc:260] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.097157 1505 tablet_bootstrap.cc:492] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:51.097381 1505 ts_tablet_manager.cc:1367] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:51.097472 1500 ts_tablet_manager.cc:1398] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:58:51.097687 1500 tablet_bootstrap.cc:492] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:51.098044 1505 raft_consensus.cc:348] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.098196 1505 raft_consensus.cc:374] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.098650 1500 tablet_bootstrap.cc:654] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.098739 1497 tablet_bootstrap.cc:492] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:51.098850 1497 ts_tablet_manager.cc:1367] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 13:58:51.098259 1505 raft_consensus.cc:729] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.099267 1497 raft_consensus.cc:348] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.099370 1497 raft_consensus.cc:374] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.099407 1497 raft_consensus.cc:729] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.099442 1505 consensus_queue.cc:260] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.099539 1497 consensus_queue.cc:260] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.099742 1505 ts_tablet_manager.cc:1398] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:51.099835 1497 ts_tablet_manager.cc:1398] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.100026 1497 tablet_bootstrap.cc:492] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:51.101347 1497 tablet_bootstrap.cc:654] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.101385 1396 raft_consensus.cc:482] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:51.101501 1500 tablet_bootstrap.cc:492] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:51.101490 1396 raft_consensus.cc:504] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.101593 1500 ts_tablet_manager.cc:1367] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:51.101953 1396 leader_election.cc:288] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:51.102187 1500 raft_consensus.cc:348] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.102393 1500 raft_consensus.cc:374] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.102471 1500 raft_consensus.cc:729] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.102555 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab75aa7826a45f6aa3de343a399c02b" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:51.102708 846 raft_consensus.cc:2454] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:51.102686 1500 consensus_queue.cc:260] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.102828 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab75aa7826a45f6aa3de343a399c02b" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:51.102921 583 raft_consensus.cc:2454] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:51.103106 1047 leader_election.cc:302] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:51.103595 1500 ts_tablet_manager.cc:1398] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:51.103767 1396 raft_consensus.cc:2781] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:51.103811 1497 tablet_bootstrap.cc:492] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:51.103835 1396 raft_consensus.cc:482] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:51.103874 1396 raft_consensus.cc:3037] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.103890 1497 ts_tablet_manager.cc:1367] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:51.104307 1497 raft_consensus.cc:348] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.104408 1497 raft_consensus.cc:374] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.104434 1497 raft_consensus.cc:729] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.104542 1497 consensus_queue.cc:260] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.104816 1497 ts_tablet_manager.cc:1398] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.107314 1333 raft_consensus.cc:482] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:51.107435 1333 raft_consensus.cc:504] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.108500 1333 leader_election.cc:288] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:51.113023 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a79b904534867bf466729be9a52a6" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:51.113266 714 raft_consensus.cc:2454] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 0.
I20240503 13:58:51.113657 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a79b904534867bf466729be9a52a6" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:51.113713 1178 leader_election.cc:302] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:51.113806 978 raft_consensus.cc:2454] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 0.
I20240503 13:58:51.114022 1333 raft_consensus.cc:2781] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:51.114115 1333 raft_consensus.cc:482] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:51.114164 1333 raft_consensus.cc:3037] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.120314 1490 raft_consensus.cc:482] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:51.120465 1490 raft_consensus.cc:504] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.121542 1490 leader_election.cc:288] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:51.122969 1396 raft_consensus.cc:504] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.123385 1396 leader_election.cc:288] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:51.123816 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab75aa7826a45f6aa3de343a399c02b" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:51.123955 583 raft_consensus.cc:3037] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.125039 583 raft_consensus.cc:2454] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:51.125694 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab75aa7826a45f6aa3de343a399c02b" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:51.125725 1047 leader_election.cc:302] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:51.125828 846 raft_consensus.cc:3037] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.126173 1396 raft_consensus.cc:2781] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:51.126385 1396 raft_consensus.cc:686] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [term 1 LEADER]: Becoming Leader. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Running, Role: LEADER
I20240503 13:58:51.126355 1333 raft_consensus.cc:504] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.126533 1396 consensus_queue.cc:237] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.126828 1333 leader_election.cc:288] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 election: Requested vote from peers 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:51.127228 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d625f94b7e5e44d1b10b6ad5f1ae9672" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:51.127347 1111 raft_consensus.cc:2454] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:51.127393 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a79b904534867bf466729be9a52a6" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:51.127504 978 raft_consensus.cc:3037] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.128149 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d625f94b7e5e44d1b10b6ad5f1ae9672" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:51.128279 583 raft_consensus.cc:2454] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:51.128396 782 leader_election.cc:302] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:51.128721 435 catalog_manager.cc:5484] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd reported cstate change: term changed from 0 to 1, leader changed from <none> to ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "ced9864d089c400e9bfaf3d07bb70acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:51.128840 1490 raft_consensus.cc:2781] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:51.128898 1490 raft_consensus.cc:482] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:51.128872 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8a79b904534867bf466729be9a52a6" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:51.128934 1490 raft_consensus.cc:3037] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.128986 714 raft_consensus.cc:3037] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.161397 714 raft_consensus.cc:2454] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 1.
I20240503 13:58:51.161397 846 raft_consensus.cc:2454] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:51.161397 978 raft_consensus.cc:2454] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 1.
I20240503 13:58:51.162009 1179 leader_election.cc:302] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2cb90442f04ba0944c9fbc78943c21, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:51.162442 1333 raft_consensus.cc:2781] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:51.162724 1333 raft_consensus.cc:686] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [term 1 LEADER]: Becoming Leader. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Running, Role: LEADER
I20240503 13:58:51.162784 1490 raft_consensus.cc:504] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.163081 1333 consensus_queue.cc:237] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:51.163163 1490 leader_election.cc:288] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Requested vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:51.163836 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d625f94b7e5e44d1b10b6ad5f1ae9672" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:51.163882 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d625f94b7e5e44d1b10b6ad5f1ae9672" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:51.163978 583 raft_consensus.cc:3037] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.163990 1111 raft_consensus.cc:3037] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.166195 435 catalog_manager.cc:5484] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d435873bcf04b3d9901363d8192342b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "7d435873bcf04b3d9901363d8192342b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.202188 583 raft_consensus.cc:2454] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:51.202188 1111 raft_consensus.cc:2454] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:51.202805 783 leader_election.cc:302] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:51.203059 1490 raft_consensus.cc:2781] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:51.203184 1490 raft_consensus.cc:686] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [term 1 LEADER]: Becoming Leader. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Running, Role: LEADER
I20240503 13:58:51.203527 1490 consensus_queue.cc:237] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:51.206231 435 catalog_manager.cc:5484] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0975c6ac192d4f90bf114279c55bd856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.330433 1521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:51.330713 1521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:51.333904 1521 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:58:51.445050 583 raft_consensus.cc:1261] T 6ab75aa7826a45f6aa3de343a399c02b P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.445645 846 raft_consensus.cc:1261] T 6ab75aa7826a45f6aa3de343a399c02b P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.446002 978 raft_consensus.cc:1261] T 8d8a79b904534867bf466729be9a52a6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 7d435873bcf04b3d9901363d8192342b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.446262 1396 consensus_queue.cc:1035] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.446604 1396 consensus_queue.cc:1035] T 6ab75aa7826a45f6aa3de343a399c02b P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.447870 1333 consensus_queue.cc:1035] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.448661 714 raft_consensus.cc:1261] T 8d8a79b904534867bf466729be9a52a6 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Refusing update from remote peer 7d435873bcf04b3d9901363d8192342b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.449942 583 raft_consensus.cc:1261] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.450464 1490 consensus_queue.cc:1035] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.452906 1111 raft_consensus.cc:1261] T d625f94b7e5e44d1b10b6ad5f1ae9672 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.454288 1490 consensus_queue.cc:1035] T d625f94b7e5e44d1b10b6ad5f1ae9672 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.455300 1333 consensus_queue.cc:1035] T 8d8a79b904534867bf466729be9a52a6 P 7d435873bcf04b3d9901363d8192342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 144.288 ms
time per row: 0.0721442 ms
I20240503 13:58:51.523540 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:51.529312 959 tablet_service.cc:1467] Processing CreateTablet for tablet 9468e574cfc44b21835df13736e6a25f (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.529367 694 tablet_service.cc:1467] Processing CreateTablet for tablet 9468e574cfc44b21835df13736e6a25f (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.529675 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9468e574cfc44b21835df13736e6a25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.529688 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9468e574cfc44b21835df13736e6a25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.530764 695 tablet_service.cc:1467] Processing CreateTablet for tablet df4a8ce3bc004b849573b3dafe24c0cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.531042 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4a8ce3bc004b849573b3dafe24c0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.531992 1501 tablet_bootstrap.cc:492] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:51.532858 1501 tablet_bootstrap.cc:654] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.533300 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 62a6523836f94087b0967fc7cab3858d (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.533602 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a6523836f94087b0967fc7cab3858d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.533707 1505 tablet_bootstrap.cc:492] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:51.534607 1505 tablet_bootstrap.cc:654] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.529335 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 9468e574cfc44b21835df13736e6a25f (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.535619 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9468e574cfc44b21835df13736e6a25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.535686 1505 tablet_bootstrap.cc:492] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:51.535761 1505 ts_tablet_manager.cc:1367] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:51.536015 1222 tablet_service.cc:1467] Processing CreateTablet for tablet df4a8ce3bc004b849573b3dafe24c0cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.536242 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4a8ce3bc004b849573b3dafe24c0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.536170 1505 raft_consensus.cc:348] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.536275 1505 raft_consensus.cc:374] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.536304 1505 raft_consensus.cc:729] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.536415 1496 tablet_bootstrap.cc:492] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:51.536422 1505 consensus_queue.cc:260] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.536696 1505 ts_tablet_manager.cc:1398] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.537261 1496 tablet_bootstrap.cc:654] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.538334 827 tablet_service.cc:1467] Processing CreateTablet for tablet 62a6523836f94087b0967fc7cab3858d (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.538434 562 tablet_service.cc:1467] Processing CreateTablet for tablet 62a6523836f94087b0967fc7cab3858d (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.538609 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a6523836f94087b0967fc7cab3858d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.538621 1498 tablet_bootstrap.cc:492] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:51.538713 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a6523836f94087b0967fc7cab3858d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.539229 1501 tablet_bootstrap.cc:492] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:51.539321 1501 ts_tablet_manager.cc:1367] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:58:51.539471 1498 tablet_bootstrap.cc:654] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.539846 1501 raft_consensus.cc:348] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.539963 1501 raft_consensus.cc:374] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.539991 1501 raft_consensus.cc:729] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.540115 1501 consensus_queue.cc:260] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.540378 1501 ts_tablet_manager.cc:1398] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.540561 1501 tablet_bootstrap.cc:492] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:51.540896 1497 tablet_bootstrap.cc:492] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:51.540892 959 tablet_service.cc:1467] Processing CreateTablet for tablet df4a8ce3bc004b849573b3dafe24c0cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=05e15f34e5df4766b0153be6cccf70b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:51.541182 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4a8ce3bc004b849573b3dafe24c0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:51.541482 1501 tablet_bootstrap.cc:654] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.541639 1500 tablet_bootstrap.cc:492] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:51.541761 1497 tablet_bootstrap.cc:654] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.542737 1500 tablet_bootstrap.cc:654] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.544332 1500 tablet_bootstrap.cc:492] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:51.544394 1501 tablet_bootstrap.cc:492] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:51.544432 1500 ts_tablet_manager.cc:1367] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:51.544698 1501 ts_tablet_manager.cc:1367] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:58:51.544886 1500 raft_consensus.cc:348] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.545001 1500 raft_consensus.cc:374] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.545032 1500 raft_consensus.cc:729] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.545101 1501 raft_consensus.cc:348] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.545202 1501 raft_consensus.cc:374] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.545173 1500 consensus_queue.cc:260] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.545233 1501 raft_consensus.cc:729] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.545379 1505 tablet_bootstrap.cc:492] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:51.545469 1500 ts_tablet_manager.cc:1398] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.545888 1501 consensus_queue.cc:260] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.545636 1498 tablet_bootstrap.cc:492] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:51.546404 1505 tablet_bootstrap.cc:654] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.546505 1501 ts_tablet_manager.cc:1398] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:51.546765 1498 ts_tablet_manager.cc:1367] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:58:51.547303 1498 raft_consensus.cc:348] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.547392 1498 raft_consensus.cc:374] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.547426 1498 raft_consensus.cc:729] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.547540 1498 consensus_queue.cc:260] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.547822 1498 ts_tablet_manager.cc:1398] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.548022 1498 tablet_bootstrap.cc:492] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:51.548074 1497 tablet_bootstrap.cc:492] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:51.548172 1497 ts_tablet_manager.cc:1367] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:58:51.548633 1497 raft_consensus.cc:348] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.548750 1497 raft_consensus.cc:374] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.548779 1497 raft_consensus.cc:729] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.548866 1498 tablet_bootstrap.cc:654] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:51.548944 1497 consensus_queue.cc:260] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.549191 1497 ts_tablet_manager.cc:1398] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:51.549917 1496 tablet_bootstrap.cc:492] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:51.550048 1496 ts_tablet_manager.cc:1367] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:58:51.550537 1496 raft_consensus.cc:348] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.550654 1496 raft_consensus.cc:374] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.550688 1496 raft_consensus.cc:729] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.550838 1505 tablet_bootstrap.cc:492] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:51.550808 1496 consensus_queue.cc:260] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.550918 1505 ts_tablet_manager.cc:1367] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:51.551067 1496 ts_tablet_manager.cc:1398] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.551175 1498 tablet_bootstrap.cc:492] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:51.551259 1498 ts_tablet_manager.cc:1367] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:51.551367 1505 raft_consensus.cc:348] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.551478 1505 raft_consensus.cc:374] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.551506 1505 raft_consensus.cc:729] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.551630 1505 consensus_queue.cc:260] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.551898 1505 ts_tablet_manager.cc:1398] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.551930 1498 raft_consensus.cc:348] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.552039 1498 raft_consensus.cc:374] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:51.552070 1498 raft_consensus.cc:729] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:51.552192 1498 consensus_queue.cc:260] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.552454 1498 ts_tablet_manager.cc:1398] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:51.566277 1516 raft_consensus.cc:482] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:51.566416 1516 raft_consensus.cc:504] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.566852 1516 leader_election.cc:288] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:51.567315 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6523836f94087b0967fc7cab3858d" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:51.567404 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6523836f94087b0967fc7cab3858d" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:51.567487 846 raft_consensus.cc:2454] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:51.567560 582 raft_consensus.cc:2454] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:51.567792 1047 leader_election.cc:302] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:51.567965 1516 raft_consensus.cc:2781] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:51.568027 1516 raft_consensus.cc:482] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:51.568071 1516 raft_consensus.cc:3037] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.612983 1516 raft_consensus.cc:504] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.613505 1516 leader_election.cc:288] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:51.614048 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6523836f94087b0967fc7cab3858d" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:51.614192 846 raft_consensus.cc:3037] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.614126 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a6523836f94087b0967fc7cab3858d" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:51.614285 582 raft_consensus.cc:3037] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.615010 846 raft_consensus.cc:2454] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:51.615213 582 raft_consensus.cc:2454] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:51.615360 1047 leader_election.cc:302] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:51.615562 1516 raft_consensus.cc:2781] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:51.615665 1516 raft_consensus.cc:686] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [term 1 LEADER]: Becoming Leader. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Running, Role: LEADER
I20240503 13:58:51.615810 1516 consensus_queue.cc:237] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:51.617755 433 catalog_manager.cc:5484] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd reported cstate change: term changed from 0 to 1, leader changed from <none> to ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "ced9864d089c400e9bfaf3d07bb70acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.626528 1468 raft_consensus.cc:482] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:51.626808 1468 raft_consensus.cc:504] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.627270 1468 leader_election.cc:288] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:51.627696 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9468e574cfc44b21835df13736e6a25f" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:51.627714 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9468e574cfc44b21835df13736e6a25f" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:51.627841 978 raft_consensus.cc:2454] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:51.627872 1243 raft_consensus.cc:2454] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:51.628261 651 leader_election.cc:302] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:51.628461 1468 raft_consensus.cc:2781] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:51.628523 1468 raft_consensus.cc:482] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:51.628549 1468 raft_consensus.cc:3037] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.646800 1471 raft_consensus.cc:482] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:51.646944 1471 raft_consensus.cc:504] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.647430 1471 leader_election.cc:288] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:51.647953 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4a8ce3bc004b849573b3dafe24c0cb" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:51.647953 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4a8ce3bc004b849573b3dafe24c0cb" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:51.648118 1243 raft_consensus.cc:2454] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:51.648111 978 raft_consensus.cc:2454] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:51.648419 649 leader_election.cc:302] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:51.648634 1471 raft_consensus.cc:2781] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:51.648698 1471 raft_consensus.cc:482] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:51.648723 1471 raft_consensus.cc:3037] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.694507 1468 raft_consensus.cc:504] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.695065 1468 leader_election.cc:288] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:51.695411 1471 raft_consensus.cc:504] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.695778 1471 leader_election.cc:288] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:51.696195 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4a8ce3bc004b849573b3dafe24c0cb" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:51.696347 1242 raft_consensus.cc:3037] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.696537 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4a8ce3bc004b849573b3dafe24c0cb" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:51.696666 979 raft_consensus.cc:3037] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.697432 1242 raft_consensus.cc:2454] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:51.697590 979 raft_consensus.cc:2454] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:51.697814 649 leader_election.cc:302] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:51.698096 1471 raft_consensus.cc:2781] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:51.698180 1471 raft_consensus.cc:686] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:51.698148 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9468e574cfc44b21835df13736e6a25f" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:51.698274 978 raft_consensus.cc:3037] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.698324 1471 consensus_queue.cc:237] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:51.698529 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9468e574cfc44b21835df13736e6a25f" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:51.698637 1243 raft_consensus.cc:3037] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:51.699156 978 raft_consensus.cc:2454] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:51.699551 1243 raft_consensus.cc:2454] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:51.699815 651 leader_election.cc:302] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:51.700134 1471 raft_consensus.cc:2781] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:51.700214 1471 raft_consensus.cc:686] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:51.700340 1471 consensus_queue.cc:237] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:51.700999 433 catalog_manager.cc:5484] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.704457 433 catalog_manager.cc:5484] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:51.854333 1573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:51.854615 1573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:51.857743 1573 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:58:51.945883 1243 raft_consensus.cc:1261] T 9468e574cfc44b21835df13736e6a25f P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.946213 978 raft_consensus.cc:1261] T 9468e574cfc44b21835df13736e6a25f P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.946720 1471 consensus_queue.cc:1035] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.947672 846 raft_consensus.cc:1261] T 62a6523836f94087b0967fc7cab3858d P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.947649 1471 consensus_queue.cc:1035] T 9468e574cfc44b21835df13736e6a25f P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.948069 582 raft_consensus.cc:1261] T 62a6523836f94087b0967fc7cab3858d P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.948683 1396 consensus_queue.cc:1035] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.949005 1396 consensus_queue.cc:1035] T 62a6523836f94087b0967fc7cab3858d P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.953951 1243 raft_consensus.cc:1261] T df4a8ce3bc004b849573b3dafe24c0cb P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.954608 1468 consensus_queue.cc:1035] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:51.955189 979 raft_consensus.cc:1261] T df4a8ce3bc004b849573b3dafe24c0cb P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:51.956396 1468 consensus_queue.cc:1035] T df4a8ce3bc004b849573b3dafe24c0cb P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.632 ms
time per row: 0.0603158 ms
I20240503 13:58:52.026628 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:52.032960 827 tablet_service.cc:1467] Processing CreateTablet for tablet 221d31f274094d88852bc8305f734b5e (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.033035 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 221d31f274094d88852bc8305f734b5e (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.033313 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221d31f274094d88852bc8305f734b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.033356 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221d31f274094d88852bc8305f734b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.033787 1222 tablet_service.cc:1467] Processing CreateTablet for tablet c800fd41f6964d588f70e57007d4038c (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.034097 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c800fd41f6964d588f70e57007d4038c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.035605 1497 tablet_bootstrap.cc:492] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:52.035905 1496 tablet_bootstrap.cc:492] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:52.036710 1497 tablet_bootstrap.cc:654] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.036897 1496 tablet_bootstrap.cc:654] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.036939 1498 tablet_bootstrap.cc:492] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:52.037356 695 tablet_service.cc:1467] Processing CreateTablet for tablet c800fd41f6964d588f70e57007d4038c (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.037672 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c800fd41f6964d588f70e57007d4038c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.037761 1498 tablet_bootstrap.cc:654] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.038094 1497 tablet_bootstrap.cc:492] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:52.038183 1497 ts_tablet_manager.cc:1367] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:58:52.038399 827 tablet_service.cc:1467] Processing CreateTablet for tablet 34a48b6c29b04743af4231ca3d595fd9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.038648 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a48b6c29b04743af4231ca3d595fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.038594 1497 raft_consensus.cc:348] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.038702 1497 raft_consensus.cc:374] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.038740 1497 raft_consensus.cc:729] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.038875 1497 consensus_queue.cc:260] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.039149 1497 ts_tablet_manager.cc:1398] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:52.039741 1501 tablet_bootstrap.cc:492] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:52.040576 1501 tablet_bootstrap.cc:654] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.040604 1497 tablet_bootstrap.cc:492] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:52.041395 959 tablet_service.cc:1467] Processing CreateTablet for tablet c800fd41f6964d588f70e57007d4038c (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.041481 1497 tablet_bootstrap.cc:654] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.041705 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c800fd41f6964d588f70e57007d4038c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.041705 562 tablet_service.cc:1467] Processing CreateTablet for tablet 221d31f274094d88852bc8305f734b5e (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.041992 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221d31f274094d88852bc8305f734b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.042939 1498 tablet_bootstrap.cc:492] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:52.043031 1498 ts_tablet_manager.cc:1367] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:58:52.043510 1498 raft_consensus.cc:348] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.043620 1498 raft_consensus.cc:374] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.043649 1498 raft_consensus.cc:729] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.043797 1498 consensus_queue.cc:260] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.044076 1498 ts_tablet_manager.cc:1398] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:52.044083 1496 tablet_bootstrap.cc:492] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:52.044150 1500 tablet_bootstrap.cc:492] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:52.044193 1496 ts_tablet_manager.cc:1367] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:58:52.044389 1505 tablet_bootstrap.cc:492] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:52.044657 1496 raft_consensus.cc:348] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.044765 1496 raft_consensus.cc:374] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.044796 1496 raft_consensus.cc:729] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.044795 1497 tablet_bootstrap.cc:492] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:52.044893 1497 ts_tablet_manager.cc:1367] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:58:52.044919 1496 consensus_queue.cc:260] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.044996 1500 tablet_bootstrap.cc:654] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.045182 1496 ts_tablet_manager.cc:1398] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.045251 1505 tablet_bootstrap.cc:654] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.045338 1497 raft_consensus.cc:348] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.045439 1497 raft_consensus.cc:374] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.045464 1497 raft_consensus.cc:729] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.045576 1497 consensus_queue.cc:260] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.044166 563 tablet_service.cc:1467] Processing CreateTablet for tablet 34a48b6c29b04743af4231ca3d595fd9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.045827 1497 ts_tablet_manager.cc:1398] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:58:52.045809 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 34a48b6c29b04743af4231ca3d595fd9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=51afcdf2ceda4886974b3bdb0c2e5676]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.046072 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a48b6c29b04743af4231ca3d595fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.045970 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a48b6c29b04743af4231ca3d595fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.047457 1505 tablet_bootstrap.cc:492] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:52.047567 1505 ts_tablet_manager.cc:1367] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:52.048064 1505 raft_consensus.cc:348] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.048166 1505 raft_consensus.cc:374] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.048197 1505 raft_consensus.cc:729] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.048332 1505 consensus_queue.cc:260] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.048619 1505 ts_tablet_manager.cc:1398] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:52.049579 1496 tablet_bootstrap.cc:492] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:52.049813 1500 tablet_bootstrap.cc:492] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:52.049896 1500 ts_tablet_manager.cc:1367] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:52.050405 1500 raft_consensus.cc:348] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.050513 1500 raft_consensus.cc:374] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.050544 1500 raft_consensus.cc:729] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.050683 1500 consensus_queue.cc:260] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.050969 1500 ts_tablet_manager.cc:1398] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.051261 1496 tablet_bootstrap.cc:654] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.051988 1500 tablet_bootstrap.cc:492] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:52.053088 1496 tablet_bootstrap.cc:492] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:52.053201 1496 ts_tablet_manager.cc:1367] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:52.053707 1500 tablet_bootstrap.cc:654] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.053710 1496 raft_consensus.cc:348] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.053824 1496 raft_consensus.cc:374] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.053856 1496 raft_consensus.cc:729] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.054039 1496 consensus_queue.cc:260] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.054343 1496 ts_tablet_manager.cc:1398] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.055191 1500 tablet_bootstrap.cc:492] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:52.055193 1501 tablet_bootstrap.cc:492] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:52.055277 1500 ts_tablet_manager.cc:1367] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:58:52.055284 1501 ts_tablet_manager.cc:1367] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 13:58:52.055706 1501 raft_consensus.cc:348] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.055810 1501 raft_consensus.cc:374] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.055847 1501 raft_consensus.cc:729] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.055979 1501 consensus_queue.cc:260] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.056257 1501 ts_tablet_manager.cc:1398] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.056726 1500 raft_consensus.cc:348] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.056839 1500 raft_consensus.cc:374] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.056872 1500 raft_consensus.cc:729] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.057025 1500 consensus_queue.cc:260] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.057307 1500 ts_tablet_manager.cc:1398] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:52.103823 1516 raft_consensus.cc:482] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:52.103966 1516 raft_consensus.cc:504] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.104374 1516 leader_election.cc:288] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:52.104858 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221d31f274094d88852bc8305f734b5e" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:52.104887 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221d31f274094d88852bc8305f734b5e" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:52.105015 846 raft_consensus.cc:2454] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:52.105028 582 raft_consensus.cc:2454] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:52.105368 1047 leader_election.cc:302] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:52.105540 1516 raft_consensus.cc:2781] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:52.105599 1516 raft_consensus.cc:482] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:52.105624 1516 raft_consensus.cc:3037] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.122566 1404 raft_consensus.cc:482] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:52.122712 1404 raft_consensus.cc:504] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.123131 1404 leader_election.cc:288] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:52.123682 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a48b6c29b04743af4231ca3d595fd9" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:52.123682 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a48b6c29b04743af4231ca3d595fd9" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:52.123854 846 raft_consensus.cc:2454] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:52.123854 1111 raft_consensus.cc:2454] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:52.124212 519 leader_election.cc:302] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:52.124468 1404 raft_consensus.cc:2781] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:52.124439 1516 raft_consensus.cc:504] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.124526 1404 raft_consensus.cc:482] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:52.124562 1404 raft_consensus.cc:3037] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.124845 1516 leader_election.cc:288] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:52.125313 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221d31f274094d88852bc8305f734b5e" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:52.125370 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221d31f274094d88852bc8305f734b5e" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:52.125433 582 raft_consensus.cc:3037] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.125487 846 raft_consensus.cc:3037] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.125636 1404 raft_consensus.cc:504] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.126034 1404 leader_election.cc:288] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:52.126363 582 raft_consensus.cc:2454] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:52.126490 846 raft_consensus.cc:2454] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:52.126560 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a48b6c29b04743af4231ca3d595fd9" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:52.126667 1111 raft_consensus.cc:3037] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.126861 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a48b6c29b04743af4231ca3d595fd9" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:52.126981 846 raft_consensus.cc:3037] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.127069 1047 leader_election.cc:302] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:52.127552 1516 raft_consensus.cc:2781] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:52.127753 1516 raft_consensus.cc:686] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [term 1 LEADER]: Becoming Leader. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Running, Role: LEADER
I20240503 13:58:52.127841 846 raft_consensus.cc:2454] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:52.127905 1516 consensus_queue.cc:237] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:52.127943 1111 raft_consensus.cc:2454] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:52.128206 519 leader_election.cc:302] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:52.128386 1404 raft_consensus.cc:2781] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:52.128520 1404 raft_consensus.cc:686] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:52.128654 1404 consensus_queue.cc:237] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:52.130522 441 catalog_manager.cc:5484] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:52.130991 439 catalog_manager.cc:5484] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd reported cstate change: term changed from 0 to 1, leader changed from <none> to ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "ced9864d089c400e9bfaf3d07bb70acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:52.183223 1335 raft_consensus.cc:482] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:52.183377 1335 raft_consensus.cc:504] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.184528 1335 leader_election.cc:288] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:52.189476 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c800fd41f6964d588f70e57007d4038c" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:52.189476 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c800fd41f6964d588f70e57007d4038c" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:52.189673 714 raft_consensus.cc:2454] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:52.189673 1242 raft_consensus.cc:2454] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:52.190112 914 leader_election.cc:302] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:52.190397 1335 raft_consensus.cc:2781] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:52.190469 1335 raft_consensus.cc:482] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:52.190507 1335 raft_consensus.cc:3037] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.218171 1335 raft_consensus.cc:504] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.218739 1335 leader_election.cc:288] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:52.219264 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c800fd41f6964d588f70e57007d4038c" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:52.219295 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c800fd41f6964d588f70e57007d4038c" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:52.219403 714 raft_consensus.cc:3037] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.219472 1242 raft_consensus.cc:3037] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.220258 714 raft_consensus.cc:2454] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:52.220467 1242 raft_consensus.cc:2454] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:52.220643 914 leader_election.cc:302] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:52.220849 1335 raft_consensus.cc:2781] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:52.221027 1335 raft_consensus.cc:686] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Running, Role: LEADER
I20240503 13:58:52.221371 1335 consensus_queue.cc:237] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:52.223922 439 catalog_manager.cc:5484] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0f2cb90442f04ba0944c9fbc78943c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:52.356887 1631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:52.357156 1631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:52.360455 1631 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 13:58:52.451572 582 raft_consensus.cc:1261] T 221d31f274094d88852bc8305f734b5e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:52.452080 1396 consensus_queue.cc:1035] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:52.452203 847 raft_consensus.cc:1261] T 34a48b6c29b04743af4231ca3d595fd9 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:52.452301 1111 raft_consensus.cc:1261] T 34a48b6c29b04743af4231ca3d595fd9 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:52.452605 1404 consensus_queue.cc:1035] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:52.452762 1623 consensus_queue.cc:1035] T 34a48b6c29b04743af4231ca3d595fd9 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:52.454032 846 raft_consensus.cc:1261] T 221d31f274094d88852bc8305f734b5e P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:52.455904 1242 raft_consensus.cc:1261] T c800fd41f6964d588f70e57007d4038c P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:52.456570 1335 consensus_queue.cc:1035] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:52.458256 1516 consensus_queue.cc:1035] T 221d31f274094d88852bc8305f734b5e P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:52.461504 714 raft_consensus.cc:1261] T c800fd41f6964d588f70e57007d4038c P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:52.462046 1335 consensus_queue.cc:1035] T c800fd41f6964d588f70e57007d4038c P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.864 ms
time per row: 0.0659319 ms
I20240503 13:58:52.544085 432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:52.550633 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 28e76f685e584257a63f79c2c69db2c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.550673 695 tablet_service.cc:1467] Processing CreateTablet for tablet 28e76f685e584257a63f79c2c69db2c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.550978 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e76f685e584257a63f79c2c69db2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.550989 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e76f685e584257a63f79c2c69db2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.551349 694 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4017f4498e4bb6a1c87ebe60ab35c7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.551417 1091 tablet_service.cc:1467] Processing CreateTablet for tablet abfdea6423ff4e7c8c1e6897cfaa6a26 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.551651 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4017f4498e4bb6a1c87ebe60ab35c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.551671 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfdea6423ff4e7c8c1e6897cfaa6a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.553251 1501 tablet_bootstrap.cc:492] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:52.553802 1496 tablet_bootstrap.cc:492] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:52.554198 1501 tablet_bootstrap.cc:654] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.554528 1671 tablet_bootstrap.cc:492] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:52.554800 1496 tablet_bootstrap.cc:654] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.554831 827 tablet_service.cc:1467] Processing CreateTablet for tablet abfdea6423ff4e7c8c1e6897cfaa6a26 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.555138 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfdea6423ff4e7c8c1e6897cfaa6a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.555712 1671 tablet_bootstrap.cc:654] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.555929 1501 tablet_bootstrap.cc:492] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:52.555860 563 tablet_service.cc:1467] Processing CreateTablet for tablet abfdea6423ff4e7c8c1e6897cfaa6a26 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.556008 1501 ts_tablet_manager.cc:1367] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:52.556190 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfdea6423ff4e7c8c1e6897cfaa6a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.556463 1501 raft_consensus.cc:348] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.556571 1501 raft_consensus.cc:374] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.556610 1501 raft_consensus.cc:729] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.556731 1501 consensus_queue.cc:260] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.556977 1501 ts_tablet_manager.cc:1398] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.557137 1501 tablet_bootstrap.cc:492] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:52.557729 1675 tablet_bootstrap.cc:492] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:52.557729 959 tablet_service.cc:1467] Processing CreateTablet for tablet 28e76f685e584257a63f79c2c69db2c6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.558053 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e76f685e584257a63f79c2c69db2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.558806 1675 tablet_bootstrap.cc:654] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.559239 1501 tablet_bootstrap.cc:654] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.559558 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4017f4498e4bb6a1c87ebe60ab35c7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.559803 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4017f4498e4bb6a1c87ebe60ab35c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.560290 1678 tablet_bootstrap.cc:492] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:52.560350 1501 tablet_bootstrap.cc:492] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:52.560532 1501 ts_tablet_manager.cc:1367] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:52.560547 958 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4017f4498e4bb6a1c87ebe60ab35c7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=62e13706ceb046668f693cd3bc839664]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:52.560823 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4017f4498e4bb6a1c87ebe60ab35c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.561100 1501 raft_consensus.cc:348] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.561208 1678 tablet_bootstrap.cc:654] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.561210 1501 raft_consensus.cc:374] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.561249 1501 raft_consensus.cc:729] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.561398 1501 consensus_queue.cc:260] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.561710 1501 ts_tablet_manager.cc:1398] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:52.562288 1500 tablet_bootstrap.cc:492] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:52.563277 1500 tablet_bootstrap.cc:654] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.563292 1678 tablet_bootstrap.cc:492] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:52.563371 1678 ts_tablet_manager.cc:1367] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:52.563755 1678 raft_consensus.cc:348] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.563838 1678 raft_consensus.cc:374] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.563856 1678 raft_consensus.cc:729] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.563962 1678 consensus_queue.cc:260] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.564224 1678 ts_tablet_manager.cc:1398] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.564383 1500 tablet_bootstrap.cc:492] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:52.564392 1678 tablet_bootstrap.cc:492] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:52.564466 1500 ts_tablet_manager.cc:1367] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:52.564883 1500 raft_consensus.cc:348] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.564996 1500 raft_consensus.cc:374] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.565028 1500 raft_consensus.cc:729] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.565224 1678 tablet_bootstrap.cc:654] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.565165 1500 consensus_queue.cc:260] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.565532 1500 ts_tablet_manager.cc:1398] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.565759 1675 tablet_bootstrap.cc:492] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:52.565850 1675 ts_tablet_manager.cc:1367] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:58:52.566342 1675 raft_consensus.cc:348] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.566453 1675 raft_consensus.cc:374] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.566493 1675 raft_consensus.cc:729] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.566615 1675 consensus_queue.cc:260] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.566866 1675 ts_tablet_manager.cc:1398] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.567265 1671 tablet_bootstrap.cc:492] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:52.567369 1671 ts_tablet_manager.cc:1367] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:58:52.567876 1671 raft_consensus.cc:348] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.567991 1671 raft_consensus.cc:374] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.568024 1671 raft_consensus.cc:729] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.568159 1671 consensus_queue.cc:260] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.568441 1671 ts_tablet_manager.cc:1398] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:52.568616 1671 tablet_bootstrap.cc:492] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:52.569068 1496 tablet_bootstrap.cc:492] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:52.569162 1496 ts_tablet_manager.cc:1367] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:58:52.569593 1496 raft_consensus.cc:348] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.569693 1496 raft_consensus.cc:374] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.569720 1496 raft_consensus.cc:729] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.571020 1496 consensus_queue.cc:260] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.571789 1496 ts_tablet_manager.cc:1398] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:52.576519 1678 tablet_bootstrap.cc:492] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:52.576802 1678 ts_tablet_manager.cc:1367] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 13:58:52.578056 1678 raft_consensus.cc:348] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.578279 1678 raft_consensus.cc:374] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.578361 1678 raft_consensus.cc:729] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.578832 1678 consensus_queue.cc:260] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.579455 1678 ts_tablet_manager.cc:1398] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:52.580864 1671 tablet_bootstrap.cc:654] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:52.582216 1671 tablet_bootstrap.cc:492] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:52.582314 1671 ts_tablet_manager.cc:1367] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 13:58:52.582788 1671 raft_consensus.cc:348] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.582963 1671 raft_consensus.cc:374] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:52.583021 1671 raft_consensus.cc:729] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:52.583163 1671 consensus_queue.cc:260] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.583505 1671 ts_tablet_manager.cc:1398] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:52.716980 1603 raft_consensus.cc:482] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:52.717113 1603 raft_consensus.cc:504] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.717509 1603 leader_election.cc:288] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:52.717938 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4017f4498e4bb6a1c87ebe60ab35c7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:52.717958 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4017f4498e4bb6a1c87ebe60ab35c7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:52.718132 1242 raft_consensus.cc:2454] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:52.718150 978 raft_consensus.cc:2454] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:52.718446 649 leader_election.cc:302] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:52.718626 1603 raft_consensus.cc:2781] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:52.718685 1603 raft_consensus.cc:482] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:52.718713 1603 raft_consensus.cc:3037] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.737746 1603 raft_consensus.cc:504] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.738332 1603 leader_election.cc:288] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:52.738790 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4017f4498e4bb6a1c87ebe60ab35c7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:52.738865 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4017f4498e4bb6a1c87ebe60ab35c7" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:52.738952 978 raft_consensus.cc:3037] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.738983 1242 raft_consensus.cc:3037] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.740047 978 raft_consensus.cc:2454] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:52.740048 1242 raft_consensus.cc:2454] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:52.740504 649 leader_election.cc:302] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:52.740722 1603 raft_consensus.cc:2781] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:52.740818 1603 raft_consensus.cc:686] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:52.740972 1603 consensus_queue.cc:237] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:52.742916 439 catalog_manager.cc:5484] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:52.750017 1404 raft_consensus.cc:482] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:52.750164 1404 raft_consensus.cc:504] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.750607 1404 leader_election.cc:288] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095), 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:52.751088 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfdea6423ff4e7c8c1e6897cfaa6a26" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:52.751111 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfdea6423ff4e7c8c1e6897cfaa6a26" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:52.751255 1111 raft_consensus.cc:2454] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:52.751256 846 raft_consensus.cc:2454] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:52.751572 519 leader_election.cc:302] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:52.751765 1404 raft_consensus.cc:2781] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:52.751822 1404 raft_consensus.cc:482] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:52.751864 1404 raft_consensus.cc:3037] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.815147 1404 raft_consensus.cc:504] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.815712 1404 leader_election.cc:288] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095), 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:52.816282 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfdea6423ff4e7c8c1e6897cfaa6a26" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:52.816308 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfdea6423ff4e7c8c1e6897cfaa6a26" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:52.816429 846 raft_consensus.cc:3037] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.816439 1111 raft_consensus.cc:3037] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.817447 846 raft_consensus.cc:2454] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:52.817468 1111 raft_consensus.cc:2454] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:52.817869 518 leader_election.cc:302] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:52.818179 1404 raft_consensus.cc:2781] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:52.818286 1404 raft_consensus.cc:686] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:52.818461 1404 consensus_queue.cc:237] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:52.820338 439 catalog_manager.cc:5484] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:52.840582 1603 raft_consensus.cc:482] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:52.840728 1603 raft_consensus.cc:504] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.841181 1603 leader_election.cc:288] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:52.841686 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e76f685e584257a63f79c2c69db2c6" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:52.841691 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e76f685e584257a63f79c2c69db2c6" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:52.841832 978 raft_consensus.cc:2454] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:52.841840 1242 raft_consensus.cc:2454] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:52.842167 651 leader_election.cc:302] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:52.842422 1683 raft_consensus.cc:2781] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:52.842484 1683 raft_consensus.cc:482] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:52.842522 1683 raft_consensus.cc:3037] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.885139 1683 raft_consensus.cc:504] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.885820 1683 leader_election.cc:288] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:52.886353 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e76f685e584257a63f79c2c69db2c6" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:52.886353 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e76f685e584257a63f79c2c69db2c6" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:52.886512 1242 raft_consensus.cc:3037] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.886534 978 raft_consensus.cc:3037] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:52.887594 978 raft_consensus.cc:2454] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:52.887594 1242 raft_consensus.cc:2454] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:52.888007 651 leader_election.cc:302] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:52.888343 1683 raft_consensus.cc:2781] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:52.888445 1683 raft_consensus.cc:686] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:52.888618 1683 consensus_queue.cc:237] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:52.890686 439 catalog_manager.cc:5484] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:53.070240 1687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:53.070547 1687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:53.074283 1687 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 13:58:53.177333 1242 raft_consensus.cc:1261] T 28e76f685e584257a63f79c2c69db2c6 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.177381 978 raft_consensus.cc:1261] T 28e76f685e584257a63f79c2c69db2c6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.177911 1603 consensus_queue.cc:1035] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.179666 1683 consensus_queue.cc:1035] T 28e76f685e584257a63f79c2c69db2c6 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.184019 1243 raft_consensus.cc:1261] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.184684 979 raft_consensus.cc:1261] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.185216 1683 consensus_queue.cc:1035] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.185933 1683 consensus_queue.cc:1035] T 7b4017f4498e4bb6a1c87ebe60ab35c7 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.201236 1111 raft_consensus.cc:1261] T abfdea6423ff4e7c8c1e6897cfaa6a26 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.201678 846 raft_consensus.cc:1261] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.202399 1404 consensus_queue.cc:1035] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.204491 1404 consensus_queue.cc:1035] T abfdea6423ff4e7c8c1e6897cfaa6a26 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.463 ms
time per row: 0.0602317 ms
I20240503 13:58:53.247931 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:53.255692 563 tablet_service.cc:1467] Processing CreateTablet for tablet 54975fd6d688494b8f7092e391695dc7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.256043 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54975fd6d688494b8f7092e391695dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.256491 562 tablet_service.cc:1467] Processing CreateTablet for tablet 868ea52451e24719b123b74f2b1732d1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.256491 694 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc9ff1f7618489ca39809f1e064cc7c (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.256749 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868ea52451e24719b123b74f2b1732d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.256752 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc9ff1f7618489ca39809f1e064cc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.258690 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 868ea52451e24719b123b74f2b1732d1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.259011 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868ea52451e24719b123b74f2b1732d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.258982 827 tablet_service.cc:1467] Processing CreateTablet for tablet 54975fd6d688494b8f7092e391695dc7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.259280 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54975fd6d688494b8f7092e391695dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.261446 1755 tablet_bootstrap.cc:492] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:53.261524 826 tablet_service.cc:1467] Processing CreateTablet for tablet 868ea52451e24719b123b74f2b1732d1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.261763 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868ea52451e24719b123b74f2b1732d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.262271 1754 tablet_bootstrap.cc:492] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:53.262421 1755 tablet_bootstrap.cc:654] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.263065 1753 tablet_bootstrap.cc:492] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:53.263114 1754 tablet_bootstrap.cc:654] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.263897 1753 tablet_bootstrap.cc:654] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.263914 958 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc9ff1f7618489ca39809f1e064cc7c (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.264156 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc9ff1f7618489ca39809f1e064cc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.264905 1755 tablet_bootstrap.cc:492] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:53.264977 1755 ts_tablet_manager.cc:1367] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:53.265434 1755 raft_consensus.cc:348] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.265535 1755 raft_consensus.cc:374] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.265564 1755 raft_consensus.cc:729] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.265718 1755 consensus_queue.cc:260] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.266000 1760 tablet_bootstrap.cc:492] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:53.266098 1755 ts_tablet_manager.cc:1398] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.266544 1753 tablet_bootstrap.cc:492] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:53.266786 1753 ts_tablet_manager.cc:1367] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:53.266901 1760 tablet_bootstrap.cc:654] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.267045 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc9ff1f7618489ca39809f1e064cc7c (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.267382 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc9ff1f7618489ca39809f1e064cc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.267347 1753 raft_consensus.cc:348] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.267455 1753 raft_consensus.cc:374] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.267486 1753 raft_consensus.cc:729] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.267597 1753 consensus_queue.cc:260] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.267949 1753 ts_tablet_manager.cc:1398] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.268119 1753 tablet_bootstrap.cc:492] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:53.268990 1753 tablet_bootstrap.cc:654] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.270246 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 54975fd6d688494b8f7092e391695dc7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7cbf9bc33a244829ae2058432e63135b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.270432 1753 tablet_bootstrap.cc:492] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:53.270493 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54975fd6d688494b8f7092e391695dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.270546 1753 ts_tablet_manager.cc:1367] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:53.270992 1753 raft_consensus.cc:348] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.271088 1753 raft_consensus.cc:374] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.271113 1753 raft_consensus.cc:729] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.271257 1753 consensus_queue.cc:260] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.271304 1762 tablet_bootstrap.cc:492] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:53.271600 1753 ts_tablet_manager.cc:1398] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.271766 1756 tablet_bootstrap.cc:492] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:53.274063 1756 tablet_bootstrap.cc:654] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.274077 1755 tablet_bootstrap.cc:492] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:53.274359 1762 tablet_bootstrap.cc:654] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.274739 1760 tablet_bootstrap.cc:492] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:53.274750 1754 tablet_bootstrap.cc:492] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:53.274842 1754 ts_tablet_manager.cc:1367] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:58:53.274840 1760 ts_tablet_manager.cc:1367] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:58:53.274904 1755 tablet_bootstrap.cc:654] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.275318 1754 raft_consensus.cc:348] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.275441 1754 raft_consensus.cc:374] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.275477 1754 raft_consensus.cc:729] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.275642 1754 consensus_queue.cc:260] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.275815 1760 raft_consensus.cc:348] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.275923 1760 raft_consensus.cc:374] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.275929 1754 ts_tablet_manager.cc:1398] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.275955 1760 raft_consensus.cc:729] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.276139 1760 consensus_queue.cc:260] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.276293 1762 tablet_bootstrap.cc:492] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:53.276432 1762 ts_tablet_manager.cc:1367] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:53.276788 1756 tablet_bootstrap.cc:492] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:53.276917 1756 ts_tablet_manager.cc:1367] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:53.276990 1755 tablet_bootstrap.cc:492] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:53.277089 1755 ts_tablet_manager.cc:1367] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:53.277069 1762 raft_consensus.cc:348] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.277170 1762 raft_consensus.cc:374] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.277199 1762 raft_consensus.cc:729] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.277344 1760 ts_tablet_manager.cc:1398] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:53.277344 1762 consensus_queue.cc:260] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.277525 1755 raft_consensus.cc:348] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.277601 1755 raft_consensus.cc:374] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.277621 1755 raft_consensus.cc:729] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.277741 1755 consensus_queue.cc:260] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.278012 1755 ts_tablet_manager.cc:1398] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:53.277388 1756 raft_consensus.cc:348] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.278204 1756 raft_consensus.cc:374] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.278237 1756 raft_consensus.cc:729] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.278357 1756 consensus_queue.cc:260] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.278578 1762 ts_tablet_manager.cc:1398] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:53.278622 1756 ts_tablet_manager.cc:1398] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:53.278800 1756 tablet_bootstrap.cc:492] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:53.279637 1756 tablet_bootstrap.cc:654] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.281244 1756 tablet_bootstrap.cc:492] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:53.281399 1756 ts_tablet_manager.cc:1367] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:58:53.282529 1756 raft_consensus.cc:348] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.282713 1756 raft_consensus.cc:374] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.282794 1756 raft_consensus.cc:729] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.282971 1756 consensus_queue.cc:260] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.283404 1756 ts_tablet_manager.cc:1398] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:53.411129 1603 raft_consensus.cc:482] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:53.411298 1603 raft_consensus.cc:504] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.411741 1603 leader_election.cc:288] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:53.412344 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc9ff1f7618489ca39809f1e064cc7c" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:53.412438 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc9ff1f7618489ca39809f1e064cc7c" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:53.412503 978 raft_consensus.cc:2454] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:53.412585 1242 raft_consensus.cc:2454] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:53.412829 651 leader_election.cc:302] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:53.413187 1742 raft_consensus.cc:2781] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:53.413246 1742 raft_consensus.cc:482] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:53.413305 1742 raft_consensus.cc:3037] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.486366 1742 raft_consensus.cc:504] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.486851 1742 leader_election.cc:288] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:53.487257 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc9ff1f7618489ca39809f1e064cc7c" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:53.487342 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc9ff1f7618489ca39809f1e064cc7c" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:53.487416 1242 raft_consensus.cc:3037] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.487466 978 raft_consensus.cc:3037] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.488479 1242 raft_consensus.cc:2454] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:53.488474 978 raft_consensus.cc:2454] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:53.488878 649 leader_election.cc:302] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:53.489148 1603 raft_consensus.cc:2781] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:53.489234 1603 raft_consensus.cc:686] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:53.489377 1603 consensus_queue.cc:237] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.491164 435 catalog_manager.cc:5484] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.535308 1490 raft_consensus.cc:482] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:53.535450 1490 raft_consensus.cc:504] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.535894 1490 leader_election.cc:288] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:53.536284 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ea52451e24719b123b74f2b1732d1" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:53.536325 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ea52451e24719b123b74f2b1732d1" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:53.536442 582 raft_consensus.cc:2454] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:53.536499 1111 raft_consensus.cc:2454] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:53.536784 782 leader_election.cc:302] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:53.536955 1490 raft_consensus.cc:2781] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:53.537029 1490 raft_consensus.cc:482] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:53.537075 1490 raft_consensus.cc:3037] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.549602 1769 raft_consensus.cc:482] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:53.549739 1769 raft_consensus.cc:504] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.550261 1769 leader_election.cc:288] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:53.550729 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54975fd6d688494b8f7092e391695dc7" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:53.550733 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54975fd6d688494b8f7092e391695dc7" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:53.550873 1111 raft_consensus.cc:2454] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:53.550890 582 raft_consensus.cc:2454] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:53.551190 782 leader_election.cc:302] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:53.551360 1769 raft_consensus.cc:2781] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:53.551424 1769 raft_consensus.cc:482] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:53.551457 1769 raft_consensus.cc:3037] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.586831 1490 raft_consensus.cc:504] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.587358 1490 leader_election.cc:288] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Requested vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:53.587761 1769 raft_consensus.cc:504] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.588104 1769 leader_election.cc:288] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Requested vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:53.588331 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ea52451e24719b123b74f2b1732d1" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:53.588469 582 raft_consensus.cc:3037] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.588594 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54975fd6d688494b8f7092e391695dc7" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:53.588593 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54975fd6d688494b8f7092e391695dc7" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:53.588728 583 raft_consensus.cc:3037] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.588721 1109 raft_consensus.cc:3037] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.589447 582 raft_consensus.cc:2454] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:53.589617 1109 raft_consensus.cc:2454] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:53.589617 583 raft_consensus.cc:2454] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:53.589803 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ea52451e24719b123b74f2b1732d1" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:53.589921 1111 raft_consensus.cc:3037] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:53.589965 782 leader_election.cc:302] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:53.590107 783 leader_election.cc:302] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:53.590176 1769 raft_consensus.cc:2781] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:53.590322 1769 raft_consensus.cc:686] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 LEADER]: Becoming Leader. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Running, Role: LEADER
I20240503 13:58:53.590343 1490 raft_consensus.cc:2781] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:53.590485 1490 raft_consensus.cc:686] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [term 1 LEADER]: Becoming Leader. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Running, Role: LEADER
I20240503 13:58:53.590462 1769 consensus_queue.cc:237] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.590827 1490 consensus_queue.cc:237] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:53.592705 435 catalog_manager.cc:5484] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0975c6ac192d4f90bf114279c55bd856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:53.592957 435 catalog_manager.cc:5484] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0975c6ac192d4f90bf114279c55bd856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:53.716745 1111 raft_consensus.cc:2454] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
W20240503 13:58:53.767432 1774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:53.767772 1774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:53.771255 1774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 13:58:53.855640 1111 raft_consensus.cc:1261] T 54975fd6d688494b8f7092e391695dc7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.856091 1490 consensus_queue.cc:1035] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.856240 583 raft_consensus.cc:1261] T 54975fd6d688494b8f7092e391695dc7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.856645 1490 consensus_queue.cc:1035] T 54975fd6d688494b8f7092e391695dc7 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.858206 978 raft_consensus.cc:1261] T 8cc9ff1f7618489ca39809f1e064cc7c P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.858670 1242 raft_consensus.cc:1261] T 8cc9ff1f7618489ca39809f1e064cc7c P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.859252 1683 consensus_queue.cc:1035] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.859465 1109 raft_consensus.cc:1261] T 868ea52451e24719b123b74f2b1732d1 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.859541 1683 consensus_queue.cc:1035] T 8cc9ff1f7618489ca39809f1e064cc7c P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.859915 1769 consensus_queue.cc:1035] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.863844 582 raft_consensus.cc:1261] T 868ea52451e24719b123b74f2b1732d1 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:53.864921 1490 consensus_queue.cc:1035] T 868ea52451e24719b123b74f2b1732d1 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:53.906000 1818 mvcc.cc:205] Tried to move back new op lower bound from 7023594429879623680 to 7023594428789297152. Current Snapshot: MvccSnapshot[applied={T|T < 7023594429879623680}]
INSERT report
rows total: 2000
time total: 105.005 ms
time per row: 0.0525023 ms
I20240503 13:58:53.925503 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:53.931921 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 85a9be66a9b0431db54452e812518546 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.932272 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a9be66a9b0431db54452e812518546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.932915 1223 tablet_service.cc:1467] Processing CreateTablet for tablet be40b8a419ee42d9a6bfea7688d35af6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.933027 958 tablet_service.cc:1467] Processing CreateTablet for tablet 85a9be66a9b0431db54452e812518546 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.933214 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40b8a419ee42d9a6bfea7688d35af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.933338 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a9be66a9b0431db54452e812518546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.934372 562 tablet_service.cc:1467] Processing CreateTablet for tablet 54ed450f8b8c4f6fbd9e4ed7909191f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.934710 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ed450f8b8c4f6fbd9e4ed7909191f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.935843 826 tablet_service.cc:1467] Processing CreateTablet for tablet 54ed450f8b8c4f6fbd9e4ed7909191f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.935843 694 tablet_service.cc:1467] Processing CreateTablet for tablet 85a9be66a9b0431db54452e812518546 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.935871 1836 tablet_bootstrap.cc:492] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:53.936095 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a9be66a9b0431db54452e812518546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.936105 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ed450f8b8c4f6fbd9e4ed7909191f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.937048 1836 tablet_bootstrap.cc:654] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.937384 959 tablet_service.cc:1467] Processing CreateTablet for tablet be40b8a419ee42d9a6bfea7688d35af6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.937664 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40b8a419ee42d9a6bfea7688d35af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.938197 1836 tablet_bootstrap.cc:492] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:53.938284 1836 ts_tablet_manager.cc:1367] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:53.938748 1836 raft_consensus.cc:348] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.938843 1841 tablet_bootstrap.cc:492] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:53.938870 1836 raft_consensus.cc:374] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.938906 1836 raft_consensus.cc:729] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.939077 1836 consensus_queue.cc:260] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.939411 1836 ts_tablet_manager.cc:1398] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.939532 1840 tablet_bootstrap.cc:492] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:53.939608 1836 tablet_bootstrap.cc:492] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:53.939776 1841 tablet_bootstrap.cc:654] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.940456 1840 tablet_bootstrap.cc:654] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.940551 1836 tablet_bootstrap.cc:654] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.940941 1841 tablet_bootstrap.cc:492] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:53.941040 1841 ts_tablet_manager.cc:1367] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:53.941290 695 tablet_service.cc:1467] Processing CreateTablet for tablet be40b8a419ee42d9a6bfea7688d35af6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.941577 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be40b8a419ee42d9a6bfea7688d35af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.941517 1841 raft_consensus.cc:348] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.941627 1841 raft_consensus.cc:374] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.941658 1841 raft_consensus.cc:729] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.941792 1841 consensus_queue.cc:260] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.942116 1841 ts_tablet_manager.cc:1398] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:58:53.942665 1837 tablet_bootstrap.cc:492] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:53.943645 1837 tablet_bootstrap.cc:654] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.944844 1837 tablet_bootstrap.cc:492] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:53.944926 1837 ts_tablet_manager.cc:1367] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:53.945107 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 54ed450f8b8c4f6fbd9e4ed7909191f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d65980ac62fe4976af554c56ce6d0776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:53.945425 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ed450f8b8c4f6fbd9e4ed7909191f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:53.945637 1840 tablet_bootstrap.cc:492] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:53.945727 1840 ts_tablet_manager.cc:1367] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:53.946324 1840 raft_consensus.cc:348] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.946424 1838 tablet_bootstrap.cc:492] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:53.946508 1840 raft_consensus.cc:374] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.946578 1836 tablet_bootstrap.cc:492] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:53.946609 1840 raft_consensus.cc:729] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.946669 1836 ts_tablet_manager.cc:1367] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:58:53.946758 1840 consensus_queue.cc:260] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.947126 1836 raft_consensus.cc:348] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.947276 1836 raft_consensus.cc:374] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.947305 1836 raft_consensus.cc:729] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.947425 1836 consensus_queue.cc:260] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.947651 1840 ts_tablet_manager.cc:1398] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:53.947676 1836 ts_tablet_manager.cc:1398] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.947813 1840 tablet_bootstrap.cc:492] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:53.948673 1840 tablet_bootstrap.cc:654] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.949241 1838 tablet_bootstrap.cc:654] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.949885 1837 raft_consensus.cc:348] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.950115 1837 raft_consensus.cc:374] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.950196 1837 raft_consensus.cc:729] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.950495 1837 consensus_queue.cc:260] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.951306 1846 tablet_bootstrap.cc:492] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:53.951862 1840 tablet_bootstrap.cc:492] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:53.952121 1840 ts_tablet_manager.cc:1367] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:53.952397 1837 ts_tablet_manager.cc:1398] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:53.952544 1846 tablet_bootstrap.cc:654] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.952656 1837 tablet_bootstrap.cc:492] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:53.952644 1840 raft_consensus.cc:348] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.952807 1840 raft_consensus.cc:374] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.952877 1840 raft_consensus.cc:729] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.953141 1840 consensus_queue.cc:260] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.953308 1838 tablet_bootstrap.cc:492] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:53.953474 1838 ts_tablet_manager.cc:1367] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 13:58:53.953486 1840 ts_tablet_manager.cc:1398] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:53.954277 1838 raft_consensus.cc:348] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.954384 1838 raft_consensus.cc:374] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.954414 1838 raft_consensus.cc:729] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.954571 1838 consensus_queue.cc:260] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.955183 1838 ts_tablet_manager.cc:1398] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:53.955494 1837 tablet_bootstrap.cc:654] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:53.957154 1837 tablet_bootstrap.cc:492] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:53.957242 1837 ts_tablet_manager.cc:1367] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:53.957757 1837 raft_consensus.cc:348] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.957882 1837 raft_consensus.cc:374] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.957914 1837 raft_consensus.cc:729] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.958168 1837 consensus_queue.cc:260] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:53.958362 1846 tablet_bootstrap.cc:492] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:53.958936 1837 ts_tablet_manager.cc:1398] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:53.959658 1846 ts_tablet_manager.cc:1367] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 13:58:53.960448 1846 raft_consensus.cc:348] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.960904 1846 raft_consensus.cc:374] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:53.960990 1846 raft_consensus.cc:729] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:53.961161 1846 consensus_queue.cc:260] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:53.961463 1846 ts_tablet_manager.cc:1398] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:54.014737 1670 raft_consensus.cc:482] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.014891 1670 raft_consensus.cc:504] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.015365 1670 leader_election.cc:288] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:54.015883 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ed450f8b8c4f6fbd9e4ed7909191f7" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:54.016037 583 raft_consensus.cc:2454] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:54.015985 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ed450f8b8c4f6fbd9e4ed7909191f7" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:54.016153 846 raft_consensus.cc:2454] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:54.016393 1047 leader_election.cc:302] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:54.016623 1670 raft_consensus.cc:2781] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.016693 1670 raft_consensus.cc:482] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.016732 1670 raft_consensus.cc:3037] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.017758 1670 raft_consensus.cc:504] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.018198 1670 leader_election.cc:288] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Requested vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:54.018690 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ed450f8b8c4f6fbd9e4ed7909191f7" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:54.018849 583 raft_consensus.cc:3037] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.018857 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ed450f8b8c4f6fbd9e4ed7909191f7" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:54.019021 846 raft_consensus.cc:3037] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.020017 846 raft_consensus.cc:2454] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:54.020021 583 raft_consensus.cc:2454] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:54.020332 1047 leader_election.cc:302] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:54.020540 1670 raft_consensus.cc:2781] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.020797 1670 raft_consensus.cc:686] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [term 1 LEADER]: Becoming Leader. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Running, Role: LEADER
I20240503 13:58:54.020948 1670 consensus_queue.cc:237] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.023027 436 catalog_manager.cc:5484] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd reported cstate change: term changed from 0 to 1, leader changed from <none> to ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "ced9864d089c400e9bfaf3d07bb70acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:54.034161 1333 raft_consensus.cc:482] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.034313 1333 raft_consensus.cc:504] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:54.034794 1333 leader_election.cc:288] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:54.035168 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a9be66a9b0431db54452e812518546" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:54.035207 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a9be66a9b0431db54452e812518546" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:54.035305 978 raft_consensus.cc:2454] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 0.
I20240503 13:58:54.035380 714 raft_consensus.cc:2454] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 0.
I20240503 13:58:54.035610 1179 leader_election.cc:302] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2cb90442f04ba0944c9fbc78943c21, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:54.035727 1700 raft_consensus.cc:482] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.035804 1333 raft_consensus.cc:2781] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.035802 1700 raft_consensus.cc:504] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:54.035866 1333 raft_consensus.cc:482] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.035907 1333 raft_consensus.cc:3037] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.036191 1700 leader_election.cc:288] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:54.036585 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40b8a419ee42d9a6bfea7688d35af6" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:54.036702 1242 raft_consensus.cc:2454] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:54.036669 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40b8a419ee42d9a6bfea7688d35af6" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:54.036798 714 raft_consensus.cc:2454] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:54.036996 913 leader_election.cc:302] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2cb90442f04ba0944c9fbc78943c21, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:54.037027 1333 raft_consensus.cc:504] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:54.037194 1700 raft_consensus.cc:2781] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.037242 1700 raft_consensus.cc:482] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.037276 1700 raft_consensus.cc:3037] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.037345 1333 leader_election.cc:288] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:54.037667 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a9be66a9b0431db54452e812518546" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:54.037675 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a9be66a9b0431db54452e812518546" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:54.037773 978 raft_consensus.cc:3037] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.037778 714 raft_consensus.cc:3037] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.038314 1700 raft_consensus.cc:504] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:54.038600 714 raft_consensus.cc:2454] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 1.
I20240503 13:58:54.038697 1700 leader_election.cc:288] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:54.038715 978 raft_consensus.cc:2454] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 1.
I20240503 13:58:54.038890 1178 leader_election.cc:302] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:54.039037 1333 raft_consensus.cc:2781] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.039111 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40b8a419ee42d9a6bfea7688d35af6" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:54.039157 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be40b8a419ee42d9a6bfea7688d35af6" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:54.039220 714 raft_consensus.cc:3037] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.039237 1333 raft_consensus.cc:686] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [term 1 LEADER]: Becoming Leader. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Running, Role: LEADER
I20240503 13:58:54.039244 1242 raft_consensus.cc:3037] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.039386 1333 consensus_queue.cc:237] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:54.040000 714 raft_consensus.cc:2454] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:54.040169 1242 raft_consensus.cc:2454] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:54.040352 914 leader_election.cc:302] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:54.040553 1700 raft_consensus.cc:2781] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.040741 1700 raft_consensus.cc:686] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Running, Role: LEADER
I20240503 13:58:54.040849 1700 consensus_queue.cc:237] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:54.041311 436 catalog_manager.cc:5484] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d435873bcf04b3d9901363d8192342b (127.0.99.134). New cstate: current_term: 1 leader_uuid: "7d435873bcf04b3d9901363d8192342b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:54.042572 439 catalog_manager.cc:5484] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0f2cb90442f04ba0944c9fbc78943c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:54.175597 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:54.175901 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:54.179369 1857 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 13:58:54.275616 846 raft_consensus.cc:1261] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.277707 714 raft_consensus.cc:1261] T be40b8a419ee42d9a6bfea7688d35af6 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.278425 1242 raft_consensus.cc:1261] T be40b8a419ee42d9a6bfea7688d35af6 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.278371 1851 consensus_queue.cc:1035] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.278900 1855 consensus_queue.cc:1035] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.279382 1700 consensus_queue.cc:1035] T be40b8a419ee42d9a6bfea7688d35af6 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.279443 583 raft_consensus.cc:1261] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.279879 1670 consensus_queue.cc:1035] T 54ed450f8b8c4f6fbd9e4ed7909191f7 P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.280472 713 raft_consensus.cc:1261] T 85a9be66a9b0431db54452e812518546 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Refusing update from remote peer 7d435873bcf04b3d9901363d8192342b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.286535 1333 consensus_queue.cc:1035] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.287402 978 raft_consensus.cc:1261] T 85a9be66a9b0431db54452e812518546 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 7d435873bcf04b3d9901363d8192342b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.292552 1853 consensus_queue.cc:1035] T 85a9be66a9b0431db54452e812518546 P 7d435873bcf04b3d9901363d8192342b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.306525 1813 mvcc.cc:205] Tried to move back new op lower bound from 7023594431608066048 to 7023594430626566144. Current Snapshot: MvccSnapshot[applied={T|T < 7023594431608066048}]
INSERT report
rows total: 2000
time total: 125.39 ms
time per row: 0.0626949 ms
I20240503 13:58:54.350981 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:54.356983 562 tablet_service.cc:1467] Processing CreateTablet for tablet d225bb01a6b1478bb88651de107ae2af (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.357101 1091 tablet_service.cc:1467] Processing CreateTablet for tablet d225bb01a6b1478bb88651de107ae2af (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.357247 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4267b7fc7d49c691a14693f7f751ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.357347 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d225bb01a6b1478bb88651de107ae2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.357424 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d225bb01a6b1478bb88651de107ae2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.357501 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4267b7fc7d49c691a14693f7f751ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.359117 695 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4267b7fc7d49c691a14693f7f751ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.359403 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4267b7fc7d49c691a14693f7f751ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.360287 1836 tablet_bootstrap.cc:492] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:54.360574 1838 tablet_bootstrap.cc:492] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:54.361196 1836 tablet_bootstrap.cc:654] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.361305 1840 tablet_bootstrap.cc:492] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:54.361510 1838 tablet_bootstrap.cc:654] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.361583 959 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4267b7fc7d49c691a14693f7f751ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.361866 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4267b7fc7d49c691a14693f7f751ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.362190 1840 tablet_bootstrap.cc:654] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.363653 1838 tablet_bootstrap.cc:492] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:54.363746 1838 ts_tablet_manager.cc:1367] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:54.363968 1836 tablet_bootstrap.cc:492] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:54.363976 1837 tablet_bootstrap.cc:492] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:54.364046 1836 ts_tablet_manager.cc:1367] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:58:54.364235 1090 tablet_service.cc:1467] Processing CreateTablet for tablet a1b3852165894867ba9a9a44e5a31b4e (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.364425 1838 raft_consensus.cc:348] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.364519 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b3852165894867ba9a9a44e5a31b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.364472 1836 raft_consensus.cc:348] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.364540 1838 raft_consensus.cc:374] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.364559 1836 raft_consensus.cc:374] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.364570 1838 raft_consensus.cc:729] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.364588 1836 raft_consensus.cc:729] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.364702 1836 consensus_queue.cc:260] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.364719 1838 consensus_queue.cc:260] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.364845 1837 tablet_bootstrap.cc:654] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.364975 1836 ts_tablet_manager.cc:1398] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:54.364984 1838 ts_tablet_manager.cc:1398] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.366261 1846 tablet_bootstrap.cc:492] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:54.367090 1846 tablet_bootstrap.cc:654] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.368230 1846 tablet_bootstrap.cc:492] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:54.368320 1846 ts_tablet_manager.cc:1367] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:54.368383 827 tablet_service.cc:1467] Processing CreateTablet for tablet a1b3852165894867ba9a9a44e5a31b4e (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.368652 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b3852165894867ba9a9a44e5a31b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.368799 1846 raft_consensus.cc:348] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.368906 1846 raft_consensus.cc:374] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.368934 1846 raft_consensus.cc:729] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.369061 1846 consensus_queue.cc:260] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.369131 562 tablet_service.cc:1467] Processing CreateTablet for tablet a1b3852165894867ba9a9a44e5a31b4e (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.369333 1846 ts_tablet_manager.cc:1398] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.369387 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b3852165894867ba9a9a44e5a31b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.369388 1840 tablet_bootstrap.cc:492] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:54.369472 1840 ts_tablet_manager.cc:1367] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:58:54.369948 1840 raft_consensus.cc:348] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.370127 1840 raft_consensus.cc:374] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.370160 1840 raft_consensus.cc:729] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.370285 1840 consensus_queue.cc:260] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.370504 1846 tablet_bootstrap.cc:492] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:54.370555 1840 ts_tablet_manager.cc:1398] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.370914 1841 tablet_bootstrap.cc:492] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:54.371639 1846 tablet_bootstrap.cc:654] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.371719 1841 tablet_bootstrap.cc:654] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.372572 1838 tablet_bootstrap.cc:492] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:54.368384 826 tablet_service.cc:1467] Processing CreateTablet for tablet d225bb01a6b1478bb88651de107ae2af (DEFAULT_TABLE table=rebalance_test_table_8 [id=55fd2739901741aa967824f001ad6968]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.373029 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d225bb01a6b1478bb88651de107ae2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.373145 1841 tablet_bootstrap.cc:492] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:54.373217 1841 ts_tablet_manager.cc:1367] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:54.373471 1838 tablet_bootstrap.cc:654] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.373631 1841 raft_consensus.cc:348] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.373741 1841 raft_consensus.cc:374] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.373769 1841 raft_consensus.cc:729] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.373891 1841 consensus_queue.cc:260] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.374325 1841 ts_tablet_manager.cc:1398] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.375044 1846 tablet_bootstrap.cc:492] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:54.375141 1846 ts_tablet_manager.cc:1367] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:54.375509 1841 tablet_bootstrap.cc:492] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:54.375612 1846 raft_consensus.cc:348] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.375734 1846 raft_consensus.cc:374] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.375771 1846 raft_consensus.cc:729] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.375898 1846 consensus_queue.cc:260] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.376206 1846 ts_tablet_manager.cc:1398] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.376372 1841 tablet_bootstrap.cc:654] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.377676 1837 tablet_bootstrap.cc:492] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:54.377784 1837 ts_tablet_manager.cc:1367] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:58:54.377811 1841 tablet_bootstrap.cc:492] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:54.377893 1841 ts_tablet_manager.cc:1367] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:54.378268 1837 raft_consensus.cc:348] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.378377 1837 raft_consensus.cc:374] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.378356 1841 raft_consensus.cc:348] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.378409 1837 raft_consensus.cc:729] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.378434 1841 raft_consensus.cc:374] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.378454 1841 raft_consensus.cc:729] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.378543 1837 consensus_queue.cc:260] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.378825 1837 ts_tablet_manager.cc:1398] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.378899 1841 consensus_queue.cc:260] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.379235 1841 ts_tablet_manager.cc:1398] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:54.398499 1838 tablet_bootstrap.cc:492] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:54.398626 1838 ts_tablet_manager.cc:1367] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 13:58:54.399057 1838 raft_consensus.cc:348] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.399180 1838 raft_consensus.cc:374] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.399212 1838 raft_consensus.cc:729] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.399339 1838 consensus_queue.cc:260] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.399639 1838 ts_tablet_manager.cc:1398] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:54.434988 1855 raft_consensus.cc:482] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.435187 1855 raft_consensus.cc:504] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.435886 1855 leader_election.cc:288] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:54.436295 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4267b7fc7d49c691a14693f7f751ed" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:54.436331 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4267b7fc7d49c691a14693f7f751ed" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:54.436451 714 raft_consensus.cc:2454] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:54.436499 1242 raft_consensus.cc:2454] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:54.436770 914 leader_election.cc:302] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:54.436987 1855 raft_consensus.cc:2781] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.437057 1855 raft_consensus.cc:482] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.437088 1855 raft_consensus.cc:3037] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.438073 1855 raft_consensus.cc:504] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.438475 1855 leader_election.cc:288] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:54.438834 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4267b7fc7d49c691a14693f7f751ed" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:54.438918 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4267b7fc7d49c691a14693f7f751ed" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:54.438997 1242 raft_consensus.cc:3037] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.439040 714 raft_consensus.cc:3037] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.439813 714 raft_consensus.cc:2454] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:54.440048 1242 raft_consensus.cc:2454] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:54.440110 914 leader_election.cc:302] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:54.440287 1855 raft_consensus.cc:2781] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.440410 1855 raft_consensus.cc:686] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Running, Role: LEADER
I20240503 13:58:54.440572 1855 consensus_queue.cc:237] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.442551 439 catalog_manager.cc:5484] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0f2cb90442f04ba0944c9fbc78943c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.453828 1769 raft_consensus.cc:482] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.453967 1769 raft_consensus.cc:504] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.454402 1769 leader_election.cc:288] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:54.454820 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b3852165894867ba9a9a44e5a31b4e" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:54.454862 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b3852165894867ba9a9a44e5a31b4e" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:54.454972 1111 raft_consensus.cc:2454] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:54.454995 583 raft_consensus.cc:2454] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:54.455286 783 leader_election.cc:302] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:54.455641 1769 raft_consensus.cc:2781] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.455694 1769 raft_consensus.cc:482] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.455719 1769 raft_consensus.cc:3037] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.456491 1769 raft_consensus.cc:504] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.456826 1769 leader_election.cc:288] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Requested vote from peers ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:54.457191 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b3852165894867ba9a9a44e5a31b4e" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:54.457206 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b3852165894867ba9a9a44e5a31b4e" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:54.457319 1111 raft_consensus.cc:3037] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.457319 583 raft_consensus.cc:3037] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.458338 1111 raft_consensus.cc:2454] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:54.458354 583 raft_consensus.cc:2454] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:54.458660 782 leader_election.cc:302] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:54.458822 1769 raft_consensus.cc:2781] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.458974 1769 raft_consensus.cc:686] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [term 1 LEADER]: Becoming Leader. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Running, Role: LEADER
I20240503 13:58:54.459124 1769 consensus_queue.cc:237] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:54.460949 439 catalog_manager.cc:5484] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0975c6ac192d4f90bf114279c55bd856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.461865 1623 raft_consensus.cc:482] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.461968 1623 raft_consensus.cc:504] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.462416 1623 leader_election.cc:288] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:54.462805 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d225bb01a6b1478bb88651de107ae2af" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:54.462853 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d225bb01a6b1478bb88651de107ae2af" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:54.462940 1111 raft_consensus.cc:2454] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:54.462978 846 raft_consensus.cc:2454] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:54.463280 519 leader_election.cc:302] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:54.463496 1623 raft_consensus.cc:2781] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.463557 1623 raft_consensus.cc:482] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.463585 1623 raft_consensus.cc:3037] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.464349 1623 raft_consensus.cc:504] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.464668 1623 leader_election.cc:288] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:54.464962 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d225bb01a6b1478bb88651de107ae2af" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:54.465001 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d225bb01a6b1478bb88651de107ae2af" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:54.465060 846 raft_consensus.cc:3037] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.465143 1111 raft_consensus.cc:3037] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.465866 1111 raft_consensus.cc:2454] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:54.466176 518 leader_election.cc:302] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:54.466331 1623 raft_consensus.cc:2781] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.466480 1623 raft_consensus.cc:686] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:54.466610 1623 consensus_queue.cc:237] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.468423 439 catalog_manager.cc:5484] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:54.482857 846 raft_consensus.cc:2454] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
W20240503 13:58:54.605563 1915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:54.605871 1915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:54.609437 1915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 13:58:54.706442 846 raft_consensus.cc:1261] T d225bb01a6b1478bb88651de107ae2af P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.706442 1111 raft_consensus.cc:1261] T d225bb01a6b1478bb88651de107ae2af P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.707026 1623 consensus_queue.cc:1035] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.707477 1623 consensus_queue.cc:1035] T d225bb01a6b1478bb88651de107ae2af P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.708729 714 raft_consensus.cc:1261] T 7b4267b7fc7d49c691a14693f7f751ed P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.709236 1855 consensus_queue.cc:1035] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.711685 1242 raft_consensus.cc:1261] T 7b4267b7fc7d49c691a14693f7f751ed P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.718330 1700 consensus_queue.cc:1035] T 7b4267b7fc7d49c691a14693f7f751ed P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.733538 1808 mvcc.cc:205] Tried to move back new op lower bound from 7023594433351585792 to 7023594432376561664. Current Snapshot: MvccSnapshot[applied={T|T < 7023594433351585792}]
I20240503 13:58:54.743578 583 raft_consensus.cc:1261] T a1b3852165894867ba9a9a44e5a31b4e P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.744110 1111 raft_consensus.cc:1261] T a1b3852165894867ba9a9a44e5a31b4e P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 0975c6ac192d4f90bf114279c55bd856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:54.744495 1769 consensus_queue.cc:1035] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:54.751024 1769 consensus_queue.cc:1035] T a1b3852165894867ba9a9a44e5a31b4e P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.244 ms
time per row: 0.0656218 ms
I20240503 13:58:54.784982 434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:54.791040 959 tablet_service.cc:1467] Processing CreateTablet for tablet 099ae452b6dc46dbaffb0a30db8a9928 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.791395 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099ae452b6dc46dbaffb0a30db8a9928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.791349 695 tablet_service.cc:1467] Processing CreateTablet for tablet 099ae452b6dc46dbaffb0a30db8a9928 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.791672 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099ae452b6dc46dbaffb0a30db8a9928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.791958 694 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb6d1cffb7f467296cfa76d75b50487 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.792163 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb6d1cffb7f467296cfa76d75b50487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.793169 1837 tablet_bootstrap.cc:492] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:54.793675 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 099ae452b6dc46dbaffb0a30db8a9928 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.793936 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099ae452b6dc46dbaffb0a30db8a9928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.794076 1840 tablet_bootstrap.cc:492] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:54.794095 1837 tablet_bootstrap.cc:654] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.794329 562 tablet_service.cc:1467] Processing CreateTablet for tablet f55b917dcc07414eb72ae7e998c0b024 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.794608 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55b917dcc07414eb72ae7e998c0b024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.794919 1840 tablet_bootstrap.cc:654] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.795652 1837 tablet_bootstrap.cc:492] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:54.795737 1837 ts_tablet_manager.cc:1367] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:58:54.796217 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb6d1cffb7f467296cfa76d75b50487 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.796187 1837 raft_consensus.cc:348] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.796314 1837 raft_consensus.cc:374] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.796342 1837 raft_consensus.cc:729] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.796447 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb6d1cffb7f467296cfa76d75b50487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.796463 1837 consensus_queue.cc:260] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.796698 1838 tablet_bootstrap.cc:492] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:54.796736 1837 ts_tablet_manager.cc:1398] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:54.796838 1840 tablet_bootstrap.cc:492] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:54.796906 1840 ts_tablet_manager.cc:1367] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:54.797122 1090 tablet_service.cc:1467] Processing CreateTablet for tablet f55b917dcc07414eb72ae7e998c0b024 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.797324 1840 raft_consensus.cc:348] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.797421 1840 raft_consensus.cc:374] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.797428 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55b917dcc07414eb72ae7e998c0b024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.797454 1840 raft_consensus.cc:729] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.797608 1838 tablet_bootstrap.cc:654] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.797564 1840 consensus_queue.cc:260] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.797827 1840 ts_tablet_manager.cc:1398] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.798593 1840 tablet_bootstrap.cc:492] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:54.798579 826 tablet_service.cc:1467] Processing CreateTablet for tablet f55b917dcc07414eb72ae7e998c0b024 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.798859 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55b917dcc07414eb72ae7e998c0b024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.799413 1840 tablet_bootstrap.cc:654] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.800007 1846 tablet_bootstrap.cc:492] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:54.800698 1838 tablet_bootstrap.cc:492] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:54.800770 1838 ts_tablet_manager.cc:1367] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:54.800806 1846 tablet_bootstrap.cc:654] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.800897 1836 tablet_bootstrap.cc:492] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:54.801196 1838 raft_consensus.cc:348] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.801301 1838 raft_consensus.cc:374] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.801328 1838 raft_consensus.cc:729] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.801445 1838 consensus_queue.cc:260] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.801680 1838 ts_tablet_manager.cc:1398] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:54.801743 1836 tablet_bootstrap.cc:654] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.802568 1841 tablet_bootstrap.cc:492] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:54.803429 1841 tablet_bootstrap.cc:654] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.805895 1841 tablet_bootstrap.cc:492] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:54.806031 1841 ts_tablet_manager.cc:1367] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:58:54.806447 1841 raft_consensus.cc:348] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.806560 1841 raft_consensus.cc:374] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.806588 1841 raft_consensus.cc:729] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.806710 1841 consensus_queue.cc:260] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.806951 1841 ts_tablet_manager.cc:1398] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.808017 959 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb6d1cffb7f467296cfa76d75b50487 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3825cfb1ac604a5d909ae86c835d4fb7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:54.808312 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb6d1cffb7f467296cfa76d75b50487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:54.812084 1840 tablet_bootstrap.cc:492] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:54.812186 1840 ts_tablet_manager.cc:1367] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:58:54.812695 1840 raft_consensus.cc:348] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.812804 1840 raft_consensus.cc:374] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.812830 1840 raft_consensus.cc:729] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.812973 1840 consensus_queue.cc:260] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.813251 1840 ts_tablet_manager.cc:1398] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.816471 1837 tablet_bootstrap.cc:492] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:54.817379 1837 tablet_bootstrap.cc:654] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.818516 1837 tablet_bootstrap.cc:492] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:54.818604 1837 ts_tablet_manager.cc:1367] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:58:54.819077 1837 raft_consensus.cc:348] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.819182 1837 raft_consensus.cc:374] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.819207 1837 raft_consensus.cc:729] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.819316 1837 consensus_queue.cc:260] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.819548 1837 ts_tablet_manager.cc:1398] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:54.819720 1846 tablet_bootstrap.cc:492] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:54.819804 1846 ts_tablet_manager.cc:1367] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240503 13:58:54.820247 1846 raft_consensus.cc:348] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.820354 1846 raft_consensus.cc:374] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.820386 1846 raft_consensus.cc:729] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.820508 1846 consensus_queue.cc:260] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:54.820787 1846 ts_tablet_manager.cc:1398] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.842438 1836 tablet_bootstrap.cc:492] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:54.842553 1836 ts_tablet_manager.cc:1367] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.002s
I20240503 13:58:54.842949 1836 raft_consensus.cc:348] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.843058 1836 raft_consensus.cc:374] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.843086 1836 raft_consensus.cc:729] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.843204 1836 consensus_queue.cc:260] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.843463 1836 ts_tablet_manager.cc:1398] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:54.843633 1836 tablet_bootstrap.cc:492] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:54.844485 1836 tablet_bootstrap.cc:654] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:54.845535 1836 tablet_bootstrap.cc:492] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:54.845628 1836 ts_tablet_manager.cc:1367] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:54.846073 1836 raft_consensus.cc:348] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.846195 1836 raft_consensus.cc:374] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:54.846226 1836 raft_consensus.cc:729] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:54.846341 1836 consensus_queue.cc:260] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:54.846809 1836 ts_tablet_manager.cc:1398] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:54.986464 1855 raft_consensus.cc:482] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:54.986621 1855 raft_consensus.cc:504] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.987061 1855 leader_election.cc:288] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:54.987566 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099ae452b6dc46dbaffb0a30db8a9928" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:54.987619 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099ae452b6dc46dbaffb0a30db8a9928" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:54.987728 714 raft_consensus.cc:2454] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:54.987803 1242 raft_consensus.cc:2454] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:54.988050 914 leader_election.cc:302] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:54.988245 1855 raft_consensus.cc:2781] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:54.988296 1855 raft_consensus.cc:482] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:54.988334 1855 raft_consensus.cc:3037] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.989176 1855 raft_consensus.cc:504] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.989544 1855 leader_election.cc:288] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Requested vote from peers 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:54.989908 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099ae452b6dc46dbaffb0a30db8a9928" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:54.990032 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099ae452b6dc46dbaffb0a30db8a9928" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:54.990075 714 raft_consensus.cc:3037] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.990137 1242 raft_consensus.cc:3037] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:54.991042 714 raft_consensus.cc:2454] T 099ae452b6dc46dbaffb0a30db8a9928 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:54.991042 1242 raft_consensus.cc:2454] T 099ae452b6dc46dbaffb0a30db8a9928 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:54.991385 914 leader_election.cc:302] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:54.991582 1855 raft_consensus.cc:2781] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:54.991665 1855 raft_consensus.cc:686] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Running, Role: LEADER
I20240503 13:58:54.991822 1855 consensus_queue.cc:237] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } }
I20240503 13:58:54.993738 435 catalog_manager.cc:5484] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0f2cb90442f04ba0944c9fbc78943c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:55.046895 1855 raft_consensus.cc:482] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:55.047039 1855 raft_consensus.cc:504] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:55.047534 1855 leader_election.cc:288] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:55.048022 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb6d1cffb7f467296cfa76d75b50487" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:55.048022 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb6d1cffb7f467296cfa76d75b50487" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:55.048192 1242 raft_consensus.cc:2454] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:55.048192 714 raft_consensus.cc:2454] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:55.048523 914 leader_election.cc:302] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:55.048725 1855 raft_consensus.cc:2781] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:55.048789 1855 raft_consensus.cc:482] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:55.048825 1855 raft_consensus.cc:3037] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.050065 1855 raft_consensus.cc:504] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:55.050417 1855 leader_election.cc:288] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:55.050756 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb6d1cffb7f467296cfa76d75b50487" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:55.050789 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb6d1cffb7f467296cfa76d75b50487" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:55.050937 714 raft_consensus.cc:3037] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.050974 1242 raft_consensus.cc:3037] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.051939 1242 raft_consensus.cc:2454] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:55.051940 714 raft_consensus.cc:2454] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:55.052268 914 leader_election.cc:302] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:55.052456 1855 raft_consensus.cc:2781] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:55.052559 1855 raft_consensus.cc:686] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Running, Role: LEADER
I20240503 13:58:55.052714 1855 consensus_queue.cc:237] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:55.054530 435 catalog_manager.cc:5484] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0f2cb90442f04ba0944c9fbc78943c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:55.219749 1950 raft_consensus.cc:482] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:55.219904 1950 raft_consensus.cc:504] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.220387 1950 leader_election.cc:288] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:55.220667 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55b917dcc07414eb72ae7e998c0b024" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:55.220832 847 raft_consensus.cc:2454] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:55.220960 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55b917dcc07414eb72ae7e998c0b024" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:55.221101 1111 raft_consensus.cc:2454] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:55.221151 519 leader_election.cc:302] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:55.221354 1950 raft_consensus.cc:2781] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:55.221450 1950 raft_consensus.cc:482] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:55.221536 1950 raft_consensus.cc:3037] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.222488 1950 raft_consensus.cc:504] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.222875 1950 leader_election.cc:288] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:55.223209 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55b917dcc07414eb72ae7e998c0b024" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:55.223318 1111 raft_consensus.cc:3037] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.224093 1111 raft_consensus.cc:2454] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:55.224344 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55b917dcc07414eb72ae7e998c0b024" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:55.224360 518 leader_election.cc:302] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:55.224459 847 raft_consensus.cc:3037] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.224563 1950 raft_consensus.cc:2781] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:55.224637 1950 raft_consensus.cc:686] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:55.224797 1950 consensus_queue.cc:237] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.225345 847 raft_consensus.cc:2454] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:55.227326 435 catalog_manager.cc:5484] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:55.384228 1966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:55.384517 1966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:55.387846 1966 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:55.410985 1700 consensus_queue.cc:1035] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:55.414875 1855 consensus_queue.cc:1035] T 099ae452b6dc46dbaffb0a30db8a9928 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 13:58:55.497144 714 raft_consensus.cc:1261] T 6cb6d1cffb7f467296cfa76d75b50487 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:55.497560 1855 consensus_queue.cc:1035] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:55.502053 1242 raft_consensus.cc:1261] T 6cb6d1cffb7f467296cfa76d75b50487 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 0f2cb90442f04ba0944c9fbc78943c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:55.502529 1700 consensus_queue.cc:1035] T 6cb6d1cffb7f467296cfa76d75b50487 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:55.505872 1111 raft_consensus.cc:1261] T f55b917dcc07414eb72ae7e998c0b024 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:55.506302 847 raft_consensus.cc:1261] T f55b917dcc07414eb72ae7e998c0b024 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer 23b6bfcaf9d148a3871c3be9ec1b41a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:55.508119 1950 consensus_queue.cc:1035] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:55.508406 1913 consensus_queue.cc:1035] T f55b917dcc07414eb72ae7e998c0b024 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:55.532099 2013 mvcc.cc:205] Tried to move back new op lower bound from 7023594436626010112 to 7023594435482271744. Current Snapshot: MvccSnapshot[applied={T|T < 7023594436626010112}]
INSERT report
rows total: 2000
time total: 111.624 ms
time per row: 0.0558122 ms
I20240503 13:58:55.556496 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:55.563369 1090 tablet_service.cc:1467] Processing CreateTablet for tablet db2747e4de1e45dbb38f1870a8f04dfc (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.563716 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2747e4de1e45dbb38f1870a8f04dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.564213 695 tablet_service.cc:1467] Processing CreateTablet for tablet d5a0fcb54b8841908a09f18d5a22e9ac (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.564260 562 tablet_service.cc:1467] Processing CreateTablet for tablet db2747e4de1e45dbb38f1870a8f04dfc (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.564512 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a0fcb54b8841908a09f18d5a22e9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.564450 563 tablet_service.cc:1467] Processing CreateTablet for tablet 4fea4b04f7e34f1284d62c7cad78242a (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.564520 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2747e4de1e45dbb38f1870a8f04dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.564702 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fea4b04f7e34f1284d62c7cad78242a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.566730 2029 tablet_bootstrap.cc:492] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:55.567065 1222 tablet_service.cc:1467] Processing CreateTablet for tablet d5a0fcb54b8841908a09f18d5a22e9ac (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.567178 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 4fea4b04f7e34f1284d62c7cad78242a (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.567178 959 tablet_service.cc:1467] Processing CreateTablet for tablet d5a0fcb54b8841908a09f18d5a22e9ac (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.567404 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a0fcb54b8841908a09f18d5a22e9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.567474 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fea4b04f7e34f1284d62c7cad78242a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.567488 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a0fcb54b8841908a09f18d5a22e9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.567786 2029 tablet_bootstrap.cc:654] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.568843 2029 tablet_bootstrap.cc:492] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:55.568922 2029 ts_tablet_manager.cc:1367] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:55.569391 2029 raft_consensus.cc:348] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.569500 2029 raft_consensus.cc:374] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.569533 2029 raft_consensus.cc:729] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.569664 2029 consensus_queue.cc:260] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.569949 2029 ts_tablet_manager.cc:1398] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.570140 2029 tablet_bootstrap.cc:492] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:55.570281 2032 tablet_bootstrap.cc:492] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:55.571075 2029 tablet_bootstrap.cc:654] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.571367 2032 tablet_bootstrap.cc:654] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.571424 2030 tablet_bootstrap.cc:492] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:55.571929 826 tablet_service.cc:1467] Processing CreateTablet for tablet db2747e4de1e45dbb38f1870a8f04dfc (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.572221 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2747e4de1e45dbb38f1870a8f04dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.572347 2030 tablet_bootstrap.cc:654] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.574478 2032 tablet_bootstrap.cc:492] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:55.574704 2036 tablet_bootstrap.cc:492] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:55.574863 827 tablet_service.cc:1467] Processing CreateTablet for tablet 4fea4b04f7e34f1284d62c7cad78242a (DEFAULT_TABLE table=rebalance_test_table_10 [id=b6c4f69df8384e9496c3cfb314f8d741]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:55.575109 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fea4b04f7e34f1284d62c7cad78242a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.575239 2030 tablet_bootstrap.cc:492] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:55.575330 2030 ts_tablet_manager.cc:1367] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:55.575620 2036 tablet_bootstrap.cc:654] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.575846 2030 raft_consensus.cc:348] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.575986 2030 raft_consensus.cc:374] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.576028 2030 raft_consensus.cc:729] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.576166 2030 consensus_queue.cc:260] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.576325 2033 tablet_bootstrap.cc:492] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:55.576444 2030 ts_tablet_manager.cc:1398] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.574575 2032 ts_tablet_manager.cc:1367] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:55.577625 2033 tablet_bootstrap.cc:654] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.578053 2032 raft_consensus.cc:348] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.578272 2032 raft_consensus.cc:374] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.578361 2032 raft_consensus.cc:729] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.578579 2032 consensus_queue.cc:260] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.578724 2029 tablet_bootstrap.cc:492] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:55.578827 2029 ts_tablet_manager.cc:1367] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 13:58:55.578877 2036 tablet_bootstrap.cc:492] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:55.578955 2036 ts_tablet_manager.cc:1367] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:58:55.579316 2029 raft_consensus.cc:348] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.579428 2029 raft_consensus.cc:374] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.579365 2036 raft_consensus.cc:348] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.579465 2036 raft_consensus.cc:374] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.579455 2029 raft_consensus.cc:729] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.579497 2036 raft_consensus.cc:729] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.579612 2029 consensus_queue.cc:260] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.579634 2036 consensus_queue.cc:260] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.579748 2032 ts_tablet_manager.cc:1398] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:55.579890 2029 ts_tablet_manager.cc:1398] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:55.579912 2036 ts_tablet_manager.cc:1398] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.580072 2036 tablet_bootstrap.cc:492] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:55.578267 2028 tablet_bootstrap.cc:492] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:55.580893 2036 tablet_bootstrap.cc:654] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.581339 2028 tablet_bootstrap.cc:654] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.582402 2036 tablet_bootstrap.cc:492] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:55.582402 2028 tablet_bootstrap.cc:492] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:55.582484 2036 ts_tablet_manager.cc:1367] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:55.582487 2028 ts_tablet_manager.cc:1367] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:55.582901 2036 raft_consensus.cc:348] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.582996 2036 raft_consensus.cc:374] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.583038 2036 raft_consensus.cc:729] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.583066 2028 raft_consensus.cc:348] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.583166 2028 raft_consensus.cc:374] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.583153 2036 consensus_queue.cc:260] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.583196 2028 raft_consensus.cc:729] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.583313 2028 consensus_queue.cc:260] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.583431 2036 ts_tablet_manager.cc:1398] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.583642 2033 tablet_bootstrap.cc:492] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:55.583675 2028 ts_tablet_manager.cc:1398] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.583747 2033 ts_tablet_manager.cc:1367] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:58:55.583889 2028 tablet_bootstrap.cc:492] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:55.584235 2033 raft_consensus.cc:348] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.584352 2033 raft_consensus.cc:374] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.584383 2033 raft_consensus.cc:729] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.584512 2033 consensus_queue.cc:260] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.584784 2033 ts_tablet_manager.cc:1398] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.584800 2028 tablet_bootstrap.cc:654] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:55.586480 2028 tablet_bootstrap.cc:492] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:55.586666 2028 ts_tablet_manager.cc:1367] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:55.587203 2028 raft_consensus.cc:348] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.587335 2028 raft_consensus.cc:374] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:55.587368 2028 raft_consensus.cc:729] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:55.587672 2028 consensus_queue.cc:260] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.588204 2028 ts_tablet_manager.cc:1398] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:55.592484 1950 raft_consensus.cc:482] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:55.592590 1950 raft_consensus.cc:504] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.592944 1950 leader_election.cc:288] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:55.593273 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea4b04f7e34f1284d62c7cad78242a" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:55.593410 847 raft_consensus.cc:2454] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:55.593398 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea4b04f7e34f1284d62c7cad78242a" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:55.593510 1111 raft_consensus.cc:2454] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 0.
I20240503 13:58:55.593684 519 leader_election.cc:302] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:55.593837 1950 raft_consensus.cc:2781] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:55.593895 1950 raft_consensus.cc:482] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:55.593937 1950 raft_consensus.cc:3037] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.594954 1950 raft_consensus.cc:504] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.595342 1950 leader_election.cc:288] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Requested vote from peers 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451), ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095)
I20240503 13:58:55.595667 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea4b04f7e34f1284d62c7cad78242a" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:55.595664 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fea4b04f7e34f1284d62c7cad78242a" candidate_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:55.595755 1111 raft_consensus.cc:3037] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.595788 847 raft_consensus.cc:3037] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.596531 1111 raft_consensus.cc:2454] T 4fea4b04f7e34f1284d62c7cad78242a P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:55.596686 847 raft_consensus.cc:2454] T 4fea4b04f7e34f1284d62c7cad78242a P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b6bfcaf9d148a3871c3be9ec1b41a0 in term 1.
I20240503 13:58:55.596786 518 leader_election.cc:302] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:55.596979 1950 raft_consensus.cc:2781] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:55.597047 1950 raft_consensus.cc:686] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 LEADER]: Becoming Leader. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Running, Role: LEADER
I20240503 13:58:55.597175 1950 consensus_queue.cc:237] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } }
I20240503 13:58:55.598996 439 catalog_manager.cc:5484] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:55.650692 2043 raft_consensus.cc:482] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:55.650837 2043 raft_consensus.cc:504] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.651266 2043 leader_election.cc:288] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:55.651723 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2747e4de1e45dbb38f1870a8f04dfc" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:55.651760 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2747e4de1e45dbb38f1870a8f04dfc" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856" is_pre_election: true
I20240503 13:58:55.651892 583 raft_consensus.cc:2454] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:55.651904 847 raft_consensus.cc:2454] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 0.
I20240503 13:58:55.652274 1047 leader_election.cc:302] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:55.652477 2043 raft_consensus.cc:2781] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:55.652537 2043 raft_consensus.cc:482] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:55.652570 2043 raft_consensus.cc:3037] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.653542 2043 raft_consensus.cc:504] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.653883 2043 leader_election.cc:288] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Requested vote from peers 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227), 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131:34451)
I20240503 13:58:55.654268 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2747e4de1e45dbb38f1870a8f04dfc" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0975c6ac192d4f90bf114279c55bd856"
I20240503 13:58:55.654285 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2747e4de1e45dbb38f1870a8f04dfc" candidate_uuid: "ced9864d089c400e9bfaf3d07bb70acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:55.654387 847 raft_consensus.cc:3037] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.654420 583 raft_consensus.cc:3037] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.655275 847 raft_consensus.cc:2454] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:55.655416 583 raft_consensus.cc:2454] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced9864d089c400e9bfaf3d07bb70acd in term 1.
I20240503 13:58:55.655709 1047 leader_election.cc:302] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b6bfcaf9d148a3871c3be9ec1b41a0, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:55.655977 2043 raft_consensus.cc:2781] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:55.656061 2043 raft_consensus.cc:686] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [term 1 LEADER]: Becoming Leader. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Running, Role: LEADER
I20240503 13:58:55.656193 2043 consensus_queue.cc:237] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } }
I20240503 13:58:55.658237 439 catalog_manager.cc:5484] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd reported cstate change: term changed from 0 to 1, leader changed from <none> to ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "ced9864d089c400e9bfaf3d07bb70acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:55.831336 1981 raft_consensus.cc:482] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:55.831482 1981 raft_consensus.cc:504] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.831912 1981 leader_election.cc:288] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:55.832382 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0fcb54b8841908a09f18d5a22e9ac" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:55.832401 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0fcb54b8841908a09f18d5a22e9ac" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:55.832525 1243 raft_consensus.cc:2454] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:55.832607 978 raft_consensus.cc:2454] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:55.832825 649 leader_election.cc:302] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:55.833000 1981 raft_consensus.cc:2781] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:55.833066 1981 raft_consensus.cc:482] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:55.833097 1981 raft_consensus.cc:3037] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.834025 1981 raft_consensus.cc:504] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.834457 1981 leader_election.cc:288] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789), 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511)
I20240503 13:58:55.834856 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0fcb54b8841908a09f18d5a22e9ac" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:55.834914 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a0fcb54b8841908a09f18d5a22e9ac" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:55.835003 1243 raft_consensus.cc:3037] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.835070 978 raft_consensus.cc:3037] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:55.835899 1243 raft_consensus.cc:2454] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:55.835968 978 raft_consensus.cc:2454] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:55.836247 649 leader_election.cc:302] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:55.836429 1981 raft_consensus.cc:2781] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:55.836521 1981 raft_consensus.cc:686] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:55.836704 1981 consensus_queue.cc:237] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } }
I20240503 13:58:55.838711 439 catalog_manager.cc:5484] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:55.989766 2047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:55.990132 2047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:55.993573 2047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:56.009740 1913 consensus_queue.cc:1035] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.015363 1950 consensus_queue.cc:1035] T 4fea4b04f7e34f1284d62c7cad78242a P 23b6bfcaf9d148a3871c3be9ec1b41a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240503 13:58:56.107354 583 raft_consensus.cc:1261] T db2747e4de1e45dbb38f1870a8f04dfc P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:56.107841 2043 consensus_queue.cc:1035] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.108522 847 raft_consensus.cc:1261] T db2747e4de1e45dbb38f1870a8f04dfc P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Refusing update from remote peer ced9864d089c400e9bfaf3d07bb70acd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:56.109743 1670 consensus_queue.cc:1035] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.111526 978 raft_consensus.cc:1261] T d5a0fcb54b8841908a09f18d5a22e9ac P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:56.111639 1243 raft_consensus.cc:1261] T d5a0fcb54b8841908a09f18d5a22e9ac P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:56.112211 1742 consensus_queue.cc:1035] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.112519 1742 consensus_queue.cc:1035] T d5a0fcb54b8841908a09f18d5a22e9ac P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.150461 2085 mvcc.cc:205] Tried to move back new op lower bound from 7023594439107936256 to 7023594437988433920. Current Snapshot: MvccSnapshot[applied={T|T < 7023594439107936256}]
I20240503 13:58:56.168118 2090 mvcc.cc:205] Tried to move back new op lower bound from 7023594439107936256 to 7023594437988433920. Current Snapshot: MvccSnapshot[applied={T|T < 7023594439107936256}]
INSERT report
rows total: 2000
time total: 134.397 ms
time per row: 0.0671985 ms
I20240503 13:58:56.180452 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56904:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:56.186715 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.186785 695 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.187088 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.187088 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.187134 694 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa698b41cb54a80905f22839e8a0f15 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.187448 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa698b41cb54a80905f22839e8a0f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.188161 958 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa698b41cb54a80905f22839e8a0f15 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.188460 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa698b41cb54a80905f22839e8a0f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.189612 959 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.189795 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 8fa698b41cb54a80905f22839e8a0f15 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.189869 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.190078 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fa698b41cb54a80905f22839e8a0f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.190318 562 tablet_service.cc:1467] Processing CreateTablet for tablet 82b73eb82934478e9c92ee688680b391 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.190624 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b73eb82934478e9c92ee688680b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.190799 2113 tablet_bootstrap.cc:492] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:56.191270 827 tablet_service.cc:1467] Processing CreateTablet for tablet 82b73eb82934478e9c92ee688680b391 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.191535 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b73eb82934478e9c92ee688680b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.193025 2114 tablet_bootstrap.cc:492] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Bootstrap starting.
I20240503 13:58:56.194228 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 82b73eb82934478e9c92ee688680b391 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05de5af91f644a15b663c7d176aa179e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:56.194396 2113 tablet_bootstrap.cc:654] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.194517 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b73eb82934478e9c92ee688680b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:56.195452 2113 tablet_bootstrap.cc:492] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:56.195542 2113 ts_tablet_manager.cc:1367] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:56.195971 2113 raft_consensus.cc:348] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.196072 2113 raft_consensus.cc:374] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.196100 2113 raft_consensus.cc:729] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.196240 2113 consensus_queue.cc:260] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.196662 2115 tablet_bootstrap.cc:492] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856: Bootstrap starting.
I20240503 13:58:56.196683 2113 ts_tablet_manager.cc:1398] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:56.196856 2113 tablet_bootstrap.cc:492] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21: Bootstrap starting.
I20240503 13:58:56.197750 2113 tablet_bootstrap.cc:654] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.199076 2114 tablet_bootstrap.cc:654] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.199077 2115 tablet_bootstrap.cc:654] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.199818 2117 tablet_bootstrap.cc:492] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd: Bootstrap starting.
I20240503 13:58:56.199818 2112 tablet_bootstrap.cc:492] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:56.200767 2117 tablet_bootstrap.cc:654] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.200817 2111 tablet_bootstrap.cc:492] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:56.201828 2114 tablet_bootstrap.cc:492] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0: No bootstrap required, opened a new log
I20240503 13:58:56.202152 2114 ts_tablet_manager.cc:1367] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:58:56.202220 2115 tablet_bootstrap.cc:492] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856: No bootstrap required, opened a new log
I20240503 13:58:56.202332 2115 ts_tablet_manager.cc:1367] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:56.202867 2115 raft_consensus.cc:348] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.202885 2114 raft_consensus.cc:348] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.202992 2114 raft_consensus.cc:374] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.203042 2114 raft_consensus.cc:729] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b6bfcaf9d148a3871c3be9ec1b41a0, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.203368 2112 tablet_bootstrap.cc:654] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.203547 2111 tablet_bootstrap.cc:654] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.203799 2114 consensus_queue.cc:260] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.204211 2114 ts_tablet_manager.cc:1398] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:56.204488 2115 raft_consensus.cc:374] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.204592 2115 raft_consensus.cc:729] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.204806 2115 consensus_queue.cc:260] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.205063 1855 raft_consensus.cc:482] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:56.205127 2117 tablet_bootstrap.cc:492] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd: No bootstrap required, opened a new log
I20240503 13:58:56.205153 1855 raft_consensus.cc:504] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.205224 2117 ts_tablet_manager.cc:1367] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:56.205538 1855 leader_election.cc:288] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:56.205731 2117 raft_consensus.cc:348] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.205837 2117 raft_consensus.cc:374] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.205868 2117 raft_consensus.cc:729] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced9864d089c400e9bfaf3d07bb70acd, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.206000 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa698b41cb54a80905f22839e8a0f15" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:56.206012 2117 consensus_queue.cc:260] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.206280 2117 ts_tablet_manager.cc:1398] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:56.206586 2115 ts_tablet_manager.cc:1398] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:58:56.207497 2111 tablet_bootstrap.cc:492] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:56.207496 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa698b41cb54a80905f22839e8a0f15" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:56.207612 2111 ts_tablet_manager.cc:1367] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
W20240503 13:58:56.207538 914 leader_election.cc:341] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551): Illegal state: must be running to vote when last-logged opid is not known
W20240503 13:58:56.208104 913 leader_election.cc:341] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:58:56.208209 913 leader_election.cc:302] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f2cb90442f04ba0944c9fbc78943c21; no voters: 02d3337c0dea4194926aa98965f3bebb, 7d435873bcf04b3d9901363d8192342b
I20240503 13:58:56.208151 2111 raft_consensus.cc:348] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.208335 2111 raft_consensus.cc:374] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.208379 2111 raft_consensus.cc:729] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.208531 2111 consensus_queue.cc:260] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.210093 2111 ts_tablet_manager.cc:1398] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:56.210106 1855 raft_consensus.cc:2726] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:58:56.210357 2111 tablet_bootstrap.cc:492] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb: Bootstrap starting.
I20240503 13:58:56.211488 2111 tablet_bootstrap.cc:654] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.213240 2112 tablet_bootstrap.cc:492] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:56.213421 2112 ts_tablet_manager.cc:1367] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:58:56.213968 2112 raft_consensus.cc:348] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.214190 2112 raft_consensus.cc:374] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.214272 2112 raft_consensus.cc:729] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.214424 2111 tablet_bootstrap.cc:492] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb: No bootstrap required, opened a new log
I20240503 13:58:56.214522 2111 ts_tablet_manager.cc:1367] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:56.214552 2112 consensus_queue.cc:260] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.214721 2113 tablet_bootstrap.cc:492] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21: No bootstrap required, opened a new log
I20240503 13:58:56.214823 2113 ts_tablet_manager.cc:1367] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240503 13:58:56.214953 2111 raft_consensus.cc:348] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.215056 2111 raft_consensus.cc:374] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.215081 2111 raft_consensus.cc:729] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.215188 2111 consensus_queue.cc:260] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.215289 2113 raft_consensus.cc:348] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.215399 2113 raft_consensus.cc:374] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.215430 2113 raft_consensus.cc:729] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.215545 2113 consensus_queue.cc:260] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.215689 2111 ts_tablet_manager.cc:1398] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:56.215816 2113 ts_tablet_manager.cc:1398] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:56.215945 2112 ts_tablet_manager.cc:1398] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:56.216110 2112 tablet_bootstrap.cc:492] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b: Bootstrap starting.
I20240503 13:58:56.217159 2112 tablet_bootstrap.cc:654] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:56.219378 2112 tablet_bootstrap.cc:492] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b: No bootstrap required, opened a new log
I20240503 13:58:56.219468 2112 ts_tablet_manager.cc:1367] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:56.219666 1333 raft_consensus.cc:482] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:56.219754 1333 raft_consensus.cc:504] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.219796 1855 raft_consensus.cc:482] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:56.219870 1855 raft_consensus.cc:504] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.220146 1333 leader_election.cc:288] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:56.220234 1855 leader_election.cc:288] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:56.220573 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:56.220701 978 raft_consensus.cc:2454] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 0.
I20240503 13:58:56.220754 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:56.220753 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:56.220857 1242 raft_consensus.cc:2454] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:56.220888 714 raft_consensus.cc:2454] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 0.
I20240503 13:58:56.220984 1179 leader_election.cc:302] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2cb90442f04ba0944c9fbc78943c21, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:56.221124 913 leader_election.cc:302] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2cb90442f04ba0944c9fbc78943c21, 7d435873bcf04b3d9901363d8192342b; no voters:
I20240503 13:58:56.221336 1333 raft_consensus.cc:2781] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:56.221386 1333 raft_consensus.cc:482] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:56.221364 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb" is_pre_election: true
I20240503 13:58:56.221419 1333 raft_consensus.cc:3037] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.221472 714 raft_consensus.cc:2454] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d435873bcf04b3d9901363d8192342b in term 0.
I20240503 13:58:56.221619 1855 raft_consensus.cc:2781] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:56.221666 1855 raft_consensus.cc:482] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:56.221699 1855 raft_consensus.cc:3037] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.222731 1855 raft_consensus.cc:504] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.223034 1333 raft_consensus.cc:504] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.223124 1855 leader_election.cc:288] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551)
I20240503 13:58:56.223363 1333 leader_election.cc:288] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 election: Requested vote from peers 02d3337c0dea4194926aa98965f3bebb (127.0.99.130:38551), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:56.223423 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:56.223470 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "0f2cb90442f04ba0944c9fbc78943c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:56.223564 714 raft_consensus.cc:3037] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.223625 1242 raft_consensus.cc:2379] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0f2cb90442f04ba0944c9fbc78943c21 in current term 1: Already voted for candidate 7d435873bcf04b3d9901363d8192342b in this term.
I20240503 13:58:56.223747 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:56.223897 978 raft_consensus.cc:2379] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d435873bcf04b3d9901363d8192342b in current term 1: Already voted for candidate 0f2cb90442f04ba0944c9fbc78943c21 in this term.
I20240503 13:58:56.224452 714 raft_consensus.cc:2454] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2cb90442f04ba0944c9fbc78943c21 in term 1.
I20240503 13:58:56.224624 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5dcadf097e4cd5bf5bfe9b95d608b2" candidate_uuid: "7d435873bcf04b3d9901363d8192342b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d3337c0dea4194926aa98965f3bebb"
I20240503 13:58:56.224691 914 leader_election.cc:302] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters: 7d435873bcf04b3d9901363d8192342b
I20240503 13:58:56.224763 713 raft_consensus.cc:2379] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d435873bcf04b3d9901363d8192342b in current term 1: Already voted for candidate 0f2cb90442f04ba0944c9fbc78943c21 in this term.
I20240503 13:58:56.224964 1855 raft_consensus.cc:2781] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:56.225039 1855 raft_consensus.cc:686] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2cb90442f04ba0944c9fbc78943c21, State: Running, Role: LEADER
I20240503 13:58:56.225082 1178 leader_election.cc:302] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d435873bcf04b3d9901363d8192342b; no voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21
I20240503 13:58:56.225168 1855 consensus_queue.cc:237] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.225395 1333 raft_consensus.cc:2726] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 13:58:56.225772 2112 raft_consensus.cc:348] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.225872 2112 raft_consensus.cc:374] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:56.225901 2112 raft_consensus.cc:729] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d435873bcf04b3d9901363d8192342b, State: Initialized, Role: FOLLOWER
I20240503 13:58:56.226063 2112 consensus_queue.cc:260] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.227077 2112 ts_tablet_manager.cc:1398] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:58:56.227396 433 catalog_manager.cc:5484] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0f2cb90442f04ba0944c9fbc78943c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:56.233886 2060 raft_consensus.cc:482] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:56.234031 2060 raft_consensus.cc:504] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.234481 2060 leader_election.cc:288] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:56.234839 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b73eb82934478e9c92ee688680b391" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd" is_pre_election: true
I20240503 13:58:56.234917 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b73eb82934478e9c92ee688680b391" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" is_pre_election: true
I20240503 13:58:56.235006 1111 raft_consensus.cc:2454] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:56.235047 583 raft_consensus.cc:2454] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 0.
I20240503 13:58:56.235345 783 leader_election.cc:302] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, 23b6bfcaf9d148a3871c3be9ec1b41a0; no voters:
I20240503 13:58:56.235522 2060 raft_consensus.cc:2781] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:56.235591 2060 raft_consensus.cc:482] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:56.235643 2060 raft_consensus.cc:3037] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.236624 2060 raft_consensus.cc:504] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.237020 2060 leader_election.cc:288] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Requested vote from peers ced9864d089c400e9bfaf3d07bb70acd (127.0.99.133:44095), 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227)
I20240503 13:58:56.237335 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b73eb82934478e9c92ee688680b391" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced9864d089c400e9bfaf3d07bb70acd"
I20240503 13:58:56.237476 1111 raft_consensus.cc:3037] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.237547 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b73eb82934478e9c92ee688680b391" candidate_uuid: "0975c6ac192d4f90bf114279c55bd856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0"
I20240503 13:58:56.237685 583 raft_consensus.cc:3037] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.238375 1111 raft_consensus.cc:2454] T 82b73eb82934478e9c92ee688680b391 P ced9864d089c400e9bfaf3d07bb70acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:56.238523 583 raft_consensus.cc:2454] T 82b73eb82934478e9c92ee688680b391 P 23b6bfcaf9d148a3871c3be9ec1b41a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0975c6ac192d4f90bf114279c55bd856 in term 1.
I20240503 13:58:56.238652 782 leader_election.cc:302] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0975c6ac192d4f90bf114279c55bd856, ced9864d089c400e9bfaf3d07bb70acd; no voters:
I20240503 13:58:56.238826 2060 raft_consensus.cc:2781] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:56.238900 2060 raft_consensus.cc:686] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [term 1 LEADER]: Becoming Leader. State: Replica: 0975c6ac192d4f90bf114279c55bd856, State: Running, Role: LEADER
I20240503 13:58:56.239030 2060 consensus_queue.cc:237] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } }
I20240503 13:58:56.241068 439 catalog_manager.cc:5484] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0975c6ac192d4f90bf114279c55bd856 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0975c6ac192d4f90bf114279c55bd856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0975c6ac192d4f90bf114279c55bd856" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:56.509042 1981 raft_consensus.cc:482] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:56.509191 1981 raft_consensus.cc:504] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.509696 1981 leader_election.cc:288] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:56.510233 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa698b41cb54a80905f22839e8a0f15" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b" is_pre_election: true
I20240503 13:58:56.510322 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa698b41cb54a80905f22839e8a0f15" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21" is_pre_election: true
I20240503 13:58:56.510425 1242 raft_consensus.cc:2454] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:56.510459 978 raft_consensus.cc:2454] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 0.
I20240503 13:58:56.510771 651 leader_election.cc:302] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:56.510953 1981 raft_consensus.cc:2781] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:56.511011 1981 raft_consensus.cc:482] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:56.511052 1981 raft_consensus.cc:3037] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.511900 1981 raft_consensus.cc:504] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.512266 1981 leader_election.cc:288] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Requested vote from peers 7d435873bcf04b3d9901363d8192342b (127.0.99.134:40511), 0f2cb90442f04ba0944c9fbc78943c21 (127.0.99.132:42789)
I20240503 13:58:56.512598 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa698b41cb54a80905f22839e8a0f15" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d435873bcf04b3d9901363d8192342b"
I20240503 13:58:56.512629 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fa698b41cb54a80905f22839e8a0f15" candidate_uuid: "02d3337c0dea4194926aa98965f3bebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2cb90442f04ba0944c9fbc78943c21"
I20240503 13:58:56.512727 979 raft_consensus.cc:3037] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.512725 1242 raft_consensus.cc:3037] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:56.513685 979 raft_consensus.cc:2454] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:56.513685 1242 raft_consensus.cc:2454] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d3337c0dea4194926aa98965f3bebb in term 1.
I20240503 13:58:56.513964 651 leader_election.cc:302] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d3337c0dea4194926aa98965f3bebb, 0f2cb90442f04ba0944c9fbc78943c21; no voters:
I20240503 13:58:56.514217 1981 raft_consensus.cc:2781] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:56.514384 1981 raft_consensus.cc:686] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [term 1 LEADER]: Becoming Leader. State: Replica: 02d3337c0dea4194926aa98965f3bebb, State: Running, Role: LEADER
I20240503 13:58:56.514544 1981 consensus_queue.cc:237] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } }
I20240503 13:58:56.516484 439 catalog_manager.cc:5484] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d3337c0dea4194926aa98965f3bebb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02d3337c0dea4194926aa98965f3bebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:56.652791 1959 consensus_queue.cc:1035] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b6bfcaf9d148a3871c3be9ec1b41a0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.656875 1959 consensus_queue.cc:1035] T 82b73eb82934478e9c92ee688680b391 P 0975c6ac192d4f90bf114279c55bd856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced9864d089c400e9bfaf3d07bb70acd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:58:56.698283 2130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:56.698552 2130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:56.701742 2130 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 13:58:56.760876 2124 consensus_queue.cc:1035] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.766471 2124 consensus_queue.cc:1035] T 3e5dcadf097e4cd5bf5bfe9b95d608b2 P 0f2cb90442f04ba0944c9fbc78943c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d3337c0dea4194926aa98965f3bebb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.788983 979 raft_consensus.cc:1261] T 8fa698b41cb54a80905f22839e8a0f15 P 0f2cb90442f04ba0944c9fbc78943c21 [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:56.789115 1242 raft_consensus.cc:1261] T 8fa698b41cb54a80905f22839e8a0f15 P 7d435873bcf04b3d9901363d8192342b [term 1 FOLLOWER]: Refusing update from remote peer 02d3337c0dea4194926aa98965f3bebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:56.790565 1742 consensus_queue.cc:1035] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d435873bcf04b3d9901363d8192342b" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:56.790908 1981 consensus_queue.cc:1035] T 8fa698b41cb54a80905f22839e8a0f15 P 02d3337c0dea4194926aa98965f3bebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2cb90442f04ba0944c9fbc78943c21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.51 ms
time per row: 0.0587551 ms
I20240503 13:58:57.100579 1091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.100687 694 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:58:57.102221 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:57.102217 827 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:58:57.105201 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.113366 1222 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.242041 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.242192 827 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:58:57.247505 1222 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.248204 1091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.257712 694 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:58:57.258939 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:57.411652 827 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:58:57.413945 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:57.415710 694 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:58:57.422053 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.435856 1091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.448402 1222 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.553763 827 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:58:57.555909 694 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:58:57.558797 1222 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.566435 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:57.569058 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.570155 1091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.691207 827 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:58:57.693235 694 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:58:57.701929 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:57.703244 1222 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.705883 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.713164 1091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.856758 1222 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:57.857102 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:57.857254 827 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:58:57.857390 694 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:58:57.859481 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:57.867460 1091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
deece84684b0459cae6e75399d829767 | 127.0.99.190:37145 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:33719 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
02d3337c0dea4194926aa98965f3bebb | 127.0.99.130:38551 | HEALTHY | /A | 10 | 0
0975c6ac192d4f90bf114279c55bd856 | 127.0.99.131:34451 | HEALTHY | /B | 5 | 0
0f2cb90442f04ba0944c9fbc78943c21 | 127.0.99.132:42789 | HEALTHY | /B | 6 | 0
23b6bfcaf9d148a3871c3be9ec1b41a0 | 127.0.99.129:37227 | HEALTHY | /A | 7 | 0
7d435873bcf04b3d9901363d8192342b | 127.0.99.134:40511 | HEALTHY | /C | 2 | 0
ced9864d089c400e9bfaf3d07bb70acd | 127.0.99.133:44095 | HEALTHY | /C | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/C | 2
/A | 2
/B | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:37227
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:38551
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:34451
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:42789
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:44095
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:40511
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:37227
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:38551
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:34451
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:42789
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:44095
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:40511
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:33719 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:58:57.906955 398 log_verifier.cc:126] Checking tablet 099ae452b6dc46dbaffb0a30db8a9928
I20240503 13:58:57.936941 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 099ae452b6dc46dbaffb0a30db8a9928
I20240503 13:58:57.937050 398 log_verifier.cc:126] Checking tablet 1188c7f43c854198b189b0110cc57f93
I20240503 13:58:57.965868 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1188c7f43c854198b189b0110cc57f93
I20240503 13:58:57.965952 398 log_verifier.cc:126] Checking tablet 221d31f274094d88852bc8305f734b5e
I20240503 13:58:57.994879 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 221d31f274094d88852bc8305f734b5e
I20240503 13:58:57.995002 398 log_verifier.cc:126] Checking tablet 28e76f685e584257a63f79c2c69db2c6
I20240503 13:58:58.024328 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 28e76f685e584257a63f79c2c69db2c6
I20240503 13:58:58.024408 398 log_verifier.cc:126] Checking tablet 34a48b6c29b04743af4231ca3d595fd9
I20240503 13:58:58.052098 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 34a48b6c29b04743af4231ca3d595fd9
I20240503 13:58:58.052176 398 log_verifier.cc:126] Checking tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2
I20240503 13:58:58.080947 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e5dcadf097e4cd5bf5bfe9b95d608b2
I20240503 13:58:58.081032 398 log_verifier.cc:126] Checking tablet 4fea4b04f7e34f1284d62c7cad78242a
I20240503 13:58:58.110595 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4fea4b04f7e34f1284d62c7cad78242a
I20240503 13:58:58.110666 398 log_verifier.cc:126] Checking tablet 54975fd6d688494b8f7092e391695dc7
I20240503 13:58:58.139940 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 54975fd6d688494b8f7092e391695dc7
I20240503 13:58:58.140010 398 log_verifier.cc:126] Checking tablet 54ed450f8b8c4f6fbd9e4ed7909191f7
I20240503 13:58:58.169719 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 54ed450f8b8c4f6fbd9e4ed7909191f7
I20240503 13:58:58.169793 398 log_verifier.cc:126] Checking tablet 62a6523836f94087b0967fc7cab3858d
I20240503 13:58:58.197928 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 62a6523836f94087b0967fc7cab3858d
I20240503 13:58:58.198032 398 log_verifier.cc:126] Checking tablet 6ab75aa7826a45f6aa3de343a399c02b
I20240503 13:58:58.226693 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6ab75aa7826a45f6aa3de343a399c02b
I20240503 13:58:58.226768 398 log_verifier.cc:126] Checking tablet 6cb6d1cffb7f467296cfa76d75b50487
I20240503 13:58:58.255718 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6cb6d1cffb7f467296cfa76d75b50487
I20240503 13:58:58.255793 398 log_verifier.cc:126] Checking tablet 746c4677ebaf4cb4bd9bd69682d423a0
I20240503 13:58:58.283890 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 746c4677ebaf4cb4bd9bd69682d423a0
I20240503 13:58:58.283967 398 log_verifier.cc:126] Checking tablet 7b4017f4498e4bb6a1c87ebe60ab35c7
I20240503 13:58:58.312597 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7b4017f4498e4bb6a1c87ebe60ab35c7
I20240503 13:58:58.312673 398 log_verifier.cc:126] Checking tablet 7b4267b7fc7d49c691a14693f7f751ed
I20240503 13:58:58.347694 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7b4267b7fc7d49c691a14693f7f751ed
I20240503 13:58:58.347787 398 log_verifier.cc:126] Checking tablet 82b73eb82934478e9c92ee688680b391
I20240503 13:58:58.378507 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 82b73eb82934478e9c92ee688680b391
I20240503 13:58:58.378578 398 log_verifier.cc:126] Checking tablet 85a9be66a9b0431db54452e812518546
I20240503 13:58:58.407341 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 85a9be66a9b0431db54452e812518546
I20240503 13:58:58.407411 398 log_verifier.cc:126] Checking tablet 868ea52451e24719b123b74f2b1732d1
I20240503 13:58:58.435564 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 868ea52451e24719b123b74f2b1732d1
I20240503 13:58:58.435629 398 log_verifier.cc:126] Checking tablet 880ae94bf80a4eaa932ca387c8a7bb51
I20240503 13:58:58.463335 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 880ae94bf80a4eaa932ca387c8a7bb51
I20240503 13:58:58.463413 398 log_verifier.cc:126] Checking tablet 8cc9ff1f7618489ca39809f1e064cc7c
I20240503 13:58:58.492007 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8cc9ff1f7618489ca39809f1e064cc7c
I20240503 13:58:58.492103 398 log_verifier.cc:126] Checking tablet 8d8a79b904534867bf466729be9a52a6
I20240503 13:58:58.519956 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8d8a79b904534867bf466729be9a52a6
I20240503 13:58:58.520030 398 log_verifier.cc:126] Checking tablet 8fa698b41cb54a80905f22839e8a0f15
I20240503 13:58:58.548741 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8fa698b41cb54a80905f22839e8a0f15
I20240503 13:58:58.548832 398 log_verifier.cc:126] Checking tablet 9468e574cfc44b21835df13736e6a25f
I20240503 13:58:58.577271 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9468e574cfc44b21835df13736e6a25f
I20240503 13:58:58.577347 398 log_verifier.cc:126] Checking tablet a1b3852165894867ba9a9a44e5a31b4e
I20240503 13:58:58.606248 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a1b3852165894867ba9a9a44e5a31b4e
I20240503 13:58:58.606314 398 log_verifier.cc:126] Checking tablet abfdea6423ff4e7c8c1e6897cfaa6a26
I20240503 13:58:58.634341 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet abfdea6423ff4e7c8c1e6897cfaa6a26
I20240503 13:58:58.634406 398 log_verifier.cc:126] Checking tablet be40b8a419ee42d9a6bfea7688d35af6
I20240503 13:58:58.662878 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet be40b8a419ee42d9a6bfea7688d35af6
I20240503 13:58:58.663040 398 log_verifier.cc:126] Checking tablet c6f32248db054f8d80934d6e6fb22454
I20240503 13:58:58.691999 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c6f32248db054f8d80934d6e6fb22454
I20240503 13:58:58.692095 398 log_verifier.cc:126] Checking tablet c800fd41f6964d588f70e57007d4038c
I20240503 13:58:58.720466 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c800fd41f6964d588f70e57007d4038c
I20240503 13:58:58.720544 398 log_verifier.cc:126] Checking tablet cadab85bb9e84c4bb2c162e7d262c681
I20240503 13:58:58.749079 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cadab85bb9e84c4bb2c162e7d262c681
I20240503 13:58:58.749150 398 log_verifier.cc:126] Checking tablet d225bb01a6b1478bb88651de107ae2af
I20240503 13:58:58.777343 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d225bb01a6b1478bb88651de107ae2af
I20240503 13:58:58.777408 398 log_verifier.cc:126] Checking tablet d5a0fcb54b8841908a09f18d5a22e9ac
I20240503 13:58:58.805186 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d5a0fcb54b8841908a09f18d5a22e9ac
I20240503 13:58:58.805263 398 log_verifier.cc:126] Checking tablet d625f94b7e5e44d1b10b6ad5f1ae9672
I20240503 13:58:58.833079 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d625f94b7e5e44d1b10b6ad5f1ae9672
I20240503 13:58:58.833149 398 log_verifier.cc:126] Checking tablet db2747e4de1e45dbb38f1870a8f04dfc
I20240503 13:58:58.861601 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet db2747e4de1e45dbb38f1870a8f04dfc
I20240503 13:58:58.861667 398 log_verifier.cc:126] Checking tablet df4a8ce3bc004b849573b3dafe24c0cb
I20240503 13:58:58.890007 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet df4a8ce3bc004b849573b3dafe24c0cb
I20240503 13:58:58.890105 398 log_verifier.cc:126] Checking tablet f55b917dcc07414eb72ae7e998c0b024
I20240503 13:58:58.918515 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f55b917dcc07414eb72ae7e998c0b024
I20240503 13:58:58.918592 398 log_verifier.cc:126] Checking tablet fe2a2afa277d4ebfa3069396de51bbb7
I20240503 13:58:58.947896 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fe2a2afa277d4ebfa3069396de51bbb7
I20240503 13:58:58.961776 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 501
W20240503 13:58:58.968698 1047 connection.cc:535] client connection to 127.0.99.129:37227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:58:58.968930 1047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:58:58.969235 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 632
W20240503 13:58:58.970846 1047 consensus_peers.cc:485] T db2747e4de1e45dbb38f1870a8f04dfc P ced9864d089c400e9bfaf3d07bb70acd -> Peer 23b6bfcaf9d148a3871c3be9ec1b41a0 (127.0.99.129:37227): Couldn't send request to peer 23b6bfcaf9d148a3871c3be9ec1b41a0. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:37227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:58.976558 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 764
I20240503 13:58:58.982816 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 897
I20240503 13:58:58.988924 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 1029
I20240503 13:58:58.996937 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 1161
I20240503 13:58:59.002816 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 410
2024-05-03T13:58:59Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (14605 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (14605 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240503 13:58:59.101187 398 test_util.cc:268] Using random seed: 521009661
I20240503 13:58:59.101958 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:58:59.102031 398 ts_itest-base.cc:113] --------------
I20240503 13:58:59.102044 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 13:58:59.102053 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:58:59.102061 398 ts_itest-base.cc:116] --------------
2024-05-03T13:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:58:59Z Disabled control of system clock
I20240503 13:58:59.110239 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34837
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34837
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 13:58:59.221354 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:59.221635 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:59.221680 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:59.224949 2482 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:59.225032 2482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:58:59.225051 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:59.225076 2482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:58:59.225107 2482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:58:59.229058 2482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34837
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:59.230119 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:59.231329 2482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:59.239784 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:59.240278 2482 server_base.cc:1034] running on GCE node
W20240503 13:58:59.240737 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:59.241684 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:59.242357 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:59.243403 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:59.244587 2482 hybrid_clock.cc:648] HybridClock initialized: now 1714744739244563 us; error 31 us; skew 500 ppm
I20240503 13:58:59.246739 2482 webserver.cc:460] Webserver started at http://127.0.99.190:35531/ using document root <none> and password file <none>
I20240503 13:58:59.247376 2482 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:59.247437 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:59.247639 2482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:59.249500 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "727855fa26de4474bd5cdf07472dff43"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.250063 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "727855fa26de4474bd5cdf07472dff43"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.253845 2482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 13:58:59.256373 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.257542 2482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:59.257687 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "727855fa26de4474bd5cdf07472dff43"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.257787 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:59.279299 2482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:59.280041 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:59.280218 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:59.286931 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34837
I20240503 13:58:59.286932 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34837 every 8 connection(s)
I20240503 13:58:59.288069 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:58:59.290850 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:59.296108 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 2482
I20240503 13:58:59.296242 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:58:59.296445 2549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43: Bootstrap starting.
I20240503 13:58:59.299008 2549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:59.299755 2549 log.cc:824] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:59.301479 2549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43: No bootstrap required, opened a new log
I20240503 13:58:59.304311 2549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727855fa26de4474bd5cdf07472dff43" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34837 } }
I20240503 13:58:59.304526 2549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:59.304577 2549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727855fa26de4474bd5cdf07472dff43, State: Initialized, Role: FOLLOWER
I20240503 13:58:59.305073 2549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727855fa26de4474bd5cdf07472dff43" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34837 } }
I20240503 13:58:59.305243 2549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:59.305306 2549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:59.305419 2549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:59.306408 2549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727855fa26de4474bd5cdf07472dff43" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34837 } }
I20240503 13:58:59.306730 2549 leader_election.cc:302] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727855fa26de4474bd5cdf07472dff43; no voters:
I20240503 13:58:59.306999 2549 leader_election.cc:288] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:59.307111 2554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:59.307365 2554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [term 1 LEADER]: Becoming Leader. State: Replica: 727855fa26de4474bd5cdf07472dff43, State: Running, Role: LEADER
I20240503 13:58:59.307651 2554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727855fa26de4474bd5cdf07472dff43" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34837 } }
I20240503 13:58:59.308138 2549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:58:59.309229 2555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727855fa26de4474bd5cdf07472dff43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727855fa26de4474bd5cdf07472dff43" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34837 } } }
I20240503 13:58:59.309253 2556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727855fa26de4474bd5cdf07472dff43. Latest consensus state: current_term: 1 leader_uuid: "727855fa26de4474bd5cdf07472dff43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727855fa26de4474bd5cdf07472dff43" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34837 } } }
I20240503 13:58:59.309569 2556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:59.309463 2555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43 [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:59.310264 2562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:58:59.312929 2562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:58:59.318248 2562 catalog_manager.cc:1347] Generated new cluster ID: 7e564999b9e4452d933817717196af66
I20240503 13:58:59.318320 2562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:58:59.331614 2562 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:58:59.332552 2562 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:58:59.339849 2562 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 727855fa26de4474bd5cdf07472dff43: Generated new TSK 0
I20240503 13:58:59.340814 2562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:58:59.353739 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--builtin_ntp_servers=127.0.99.148:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:58:59.467907 2573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:59.468179 2573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:59.468248 2573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:59.471518 2573 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:59.471616 2573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:59.471711 2573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:58:59.475589 2573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--never_fsync=true
--heap_profile_path=/tmp/kudu.2573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:59.476606 2573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:59.477833 2573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:59.487046 2579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:59.487540 2578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:59.488485 2581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:59.489178 2573 server_base.cc:1034] running on GCE node
I20240503 13:58:59.489698 2573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:59.490455 2573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:59.491624 2573 hybrid_clock.cc:648] HybridClock initialized: now 1714744739491612 us; error 29 us; skew 500 ppm
I20240503 13:58:59.493795 2573 webserver.cc:460] Webserver started at http://127.0.99.129:39005/ using document root <none> and password file <none>
I20240503 13:58:59.494426 2573 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:59.494486 2573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:59.494678 2573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:59.496460 2573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3746179b951d4ce1b3e5680a250eadeb"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.496968 2573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3746179b951d4ce1b3e5680a250eadeb"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.500535 2573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:58:59.502991 2587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.504062 2573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:59.504173 2573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3746179b951d4ce1b3e5680a250eadeb"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.504277 2573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:59.531482 2573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:59.532282 2573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:59.532454 2573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:59.532990 2573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:59.533921 2573 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:59.533969 2573 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.534044 2573 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:59.534070 2573 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.542402 2573 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:40247
I20240503 13:58:59.542431 2700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:40247 every 8 connection(s)
I20240503 13:58:59.543424 2573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:58:59.550246 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 2573
I20240503 13:58:59.550347 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:58:59.552516 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--builtin_ntp_servers=127.0.99.148:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:59.552672 2701 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34837
I20240503 13:58:59.552907 2701 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:59.553490 2701 heartbeater.cc:511] Master 127.0.99.190:34837 requested a full tablet report, sending...
I20240503 13:58:59.555147 2513 ts_manager.cc:194] Registered new tserver with Master: 3746179b951d4ce1b3e5680a250eadeb (127.0.99.129:40247)
I20240503 13:58:59.556828 2513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:46863
W20240503 13:58:59.665591 2704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:59.665876 2704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:59.665925 2704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:59.669277 2704 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:59.669376 2704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:59.669450 2704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:58:59.673352 2704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--never_fsync=true
--heap_profile_path=/tmp/kudu.2704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:59.674386 2704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:59.675626 2704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:59.685335 2709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:59.686070 2712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:59.686707 2704 server_base.cc:1034] running on GCE node
W20240503 13:58:59.687108 2710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:59.687599 2704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:59.688287 2704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:59.689447 2704 hybrid_clock.cc:648] HybridClock initialized: now 1714744739689429 us; error 34 us; skew 500 ppm
I20240503 13:58:59.691598 2704 webserver.cc:460] Webserver started at http://127.0.99.130:43659/ using document root <none> and password file <none>
I20240503 13:58:59.692217 2704 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:59.692282 2704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:59.692468 2704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:59.694343 2704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d81a247057194f2ea0771201bbe827ad"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.694866 2704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d81a247057194f2ea0771201bbe827ad"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.698453 2704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:59.700865 2718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.702032 2704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:59.702152 2704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d81a247057194f2ea0771201bbe827ad"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.702261 2704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:59.718693 2704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:59.719441 2704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:59.719652 2704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:59.720274 2704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:59.721452 2704 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:59.721520 2704 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.721622 2704 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:59.721660 2704 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.733177 2704 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33767
I20240503 13:58:59.733215 2831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33767 every 8 connection(s)
I20240503 13:58:59.734587 2704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:58:59.743697 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 2704
I20240503 13:58:59.743813 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:58:59.744385 2832 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34837
I20240503 13:58:59.744678 2832 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:59.745321 2832 heartbeater.cc:511] Master 127.0.99.190:34837 requested a full tablet report, sending...
I20240503 13:58:59.746052 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--builtin_ntp_servers=127.0.99.148:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:59.746452 2513 ts_manager.cc:194] Registered new tserver with Master: d81a247057194f2ea0771201bbe827ad (127.0.99.130:33767)
I20240503 13:58:59.747028 2513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:46087
W20240503 13:58:59.859642 2835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:59.859921 2835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:59.859969 2835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:59.863262 2835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:59.863341 2835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:59.863428 2835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:59.867220 2835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--never_fsync=true
--heap_profile_path=/tmp/kudu.2835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:59.868245 2835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:59.869421 2835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:59.878371 2840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:59.878954 2843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:59.879747 2841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:59.880471 2835 server_base.cc:1034] running on GCE node
I20240503 13:58:59.880921 2835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:59.881572 2835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:59.882725 2835 hybrid_clock.cc:648] HybridClock initialized: now 1714744739882709 us; error 41 us; skew 500 ppm
I20240503 13:58:59.884846 2835 webserver.cc:460] Webserver started at http://127.0.99.131:40009/ using document root <none> and password file <none>
I20240503 13:58:59.885437 2835 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:59.885496 2835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:59.885675 2835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:59.887490 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0e9434614026420c8fbda104a9dc89b1"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.887964 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0e9434614026420c8fbda104a9dc89b1"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.891481 2835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:59.893756 2849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.894834 2835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:59.894946 2835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0e9434614026420c8fbda104a9dc89b1"
format_stamp: "Formatted at 2024-05-03 13:58:59 on dist-test-slave-6fwv"
I20240503 13:58:59.895037 2835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:59.928580 2835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:59.929301 2835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:59.929484 2835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:59.930115 2835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:59.931178 2835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:59.931231 2835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.931288 2835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:59.931315 2835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:59.940286 2835 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:42259
I20240503 13:58:59.940305 2962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:42259 every 8 connection(s)
I20240503 13:58:59.941337 2835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:59.942111 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 2835
I20240503 13:58:59.942204 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:58:59.944711 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--builtin_ntp_servers=127.0.99.148:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:58:59.951535 2963 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34837
I20240503 13:58:59.951797 2963 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:59.952405 2963 heartbeater.cc:511] Master 127.0.99.190:34837 requested a full tablet report, sending...
I20240503 13:58:59.953522 2513 ts_manager.cc:194] Registered new tserver with Master: 0e9434614026420c8fbda104a9dc89b1 (127.0.99.131:42259)
I20240503 13:58:59.954146 2513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:59003
W20240503 13:59:00.061152 2966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:00.061432 2966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:00.061498 2966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:00.064916 2966 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:00.065011 2966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:00.065109 2966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:00.069257 2966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--never_fsync=true
--heap_profile_path=/tmp/kudu.2966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:00.070397 2966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:00.071625 2966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:00.080868 2971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:00.081319 2972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:00.082163 2974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:00.082896 2966 server_base.cc:1034] running on GCE node
I20240503 13:59:00.083395 2966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:00.084062 2966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:00.085211 2966 hybrid_clock.cc:648] HybridClock initialized: now 1714744740085182 us; error 47 us; skew 500 ppm
I20240503 13:59:00.087445 2966 webserver.cc:460] Webserver started at http://127.0.99.132:35621/ using document root <none> and password file <none>
I20240503 13:59:00.088063 2966 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:00.088126 2966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:00.088310 2966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:00.090176 2966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ed90aa0238274b3195b48db10fde8333"
format_stamp: "Formatted at 2024-05-03 13:59:00 on dist-test-slave-6fwv"
I20240503 13:59:00.090675 2966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ed90aa0238274b3195b48db10fde8333"
format_stamp: "Formatted at 2024-05-03 13:59:00 on dist-test-slave-6fwv"
I20240503 13:59:00.094242 2966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:00.096607 2980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:00.097721 2966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:00.097841 2966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ed90aa0238274b3195b48db10fde8333"
format_stamp: "Formatted at 2024-05-03 13:59:00 on dist-test-slave-6fwv"
I20240503 13:59:00.097944 2966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:00.125228 2966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:00.125947 2966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:00.126166 2966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:00.126750 2966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:00.127755 2966 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:00.127805 2966 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:00.127863 2966 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:00.127890 2966 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:00.136569 2966 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37547
I20240503 13:59:00.136593 3093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37547 every 8 connection(s)
I20240503 13:59:00.137593 2966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:00.140940 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 2966
I20240503 13:59:00.141036 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:00.143234 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--builtin_ntp_servers=127.0.99.148:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:00.147233 3094 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34837
I20240503 13:59:00.147502 3094 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:00.148123 3094 heartbeater.cc:511] Master 127.0.99.190:34837 requested a full tablet report, sending...
I20240503 13:59:00.149241 2513 ts_manager.cc:194] Registered new tserver with Master: ed90aa0238274b3195b48db10fde8333 (127.0.99.132:37547)
I20240503 13:59:00.150063 2513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55575
W20240503 13:59:00.256564 3097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:00.256842 3097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:00.256889 3097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:00.260179 3097 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:00.260268 3097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:00.260342 3097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:00.264314 3097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34837
--never_fsync=true
--heap_profile_path=/tmp/kudu.3097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:00.265304 3097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:00.266522 3097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:00.276044 3102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:00.276744 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:00.277292 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:00.277966 3097 server_base.cc:1034] running on GCE node
I20240503 13:59:00.278468 3097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:00.279187 3097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:00.280375 3097 hybrid_clock.cc:648] HybridClock initialized: now 1714744740280357 us; error 37 us; skew 500 ppm
I20240503 13:59:00.282644 3097 webserver.cc:460] Webserver started at http://127.0.99.133:37147/ using document root <none> and password file <none>
I20240503 13:59:00.283263 3097 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:00.283324 3097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:00.283504 3097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:00.285347 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b5682dac660e4f1c9d6b3b14f5315e01"
format_stamp: "Formatted at 2024-05-03 13:59:00 on dist-test-slave-6fwv"
I20240503 13:59:00.285835 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b5682dac660e4f1c9d6b3b14f5315e01"
format_stamp: "Formatted at 2024-05-03 13:59:00 on dist-test-slave-6fwv"
I20240503 13:59:00.290097 3097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:00.292943 3111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:00.294286 3097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:00.294415 3097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b5682dac660e4f1c9d6b3b14f5315e01"
format_stamp: "Formatted at 2024-05-03 13:59:00 on dist-test-slave-6fwv"
I20240503 13:59:00.294521 3097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:00.312691 3097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:00.313406 3097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:00.313602 3097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:00.314213 3097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:00.315155 3097 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:00.315202 3097 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:00.315241 3097 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:00.315254 3097 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:00.323477 3097 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34453
I20240503 13:59:00.323498 3224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34453 every 8 connection(s)
I20240503 13:59:00.324507 3097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:00.329959 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 3097
I20240503 13:59:00.330119 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:00.334041 3225 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34837
I20240503 13:59:00.334309 3225 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:00.334904 3225 heartbeater.cc:511] Master 127.0.99.190:34837 requested a full tablet report, sending...
I20240503 13:59:00.335965 2513 ts_manager.cc:194] Registered new tserver with Master: b5682dac660e4f1c9d6b3b14f5315e01 (127.0.99.133:34453)
I20240503 13:59:00.336534 2513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:42409
I20240503 13:59:00.344570 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:59:00.353801 2513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39696:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:59:00.372615 3028 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1f974c41e8497490ee4662d5f6364a (DEFAULT_TABLE table=TestTable [id=daf71a93353d48a082b88b317a91acba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:00.373217 3159 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1f974c41e8497490ee4662d5f6364a (DEFAULT_TABLE table=TestTable [id=daf71a93353d48a082b88b317a91acba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:00.373752 3028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1f974c41e8497490ee4662d5f6364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:00.374063 3159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1f974c41e8497490ee4662d5f6364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:00.374365 2635 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1f974c41e8497490ee4662d5f6364a (DEFAULT_TABLE table=TestTable [id=daf71a93353d48a082b88b317a91acba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:00.375257 2635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1f974c41e8497490ee4662d5f6364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:00.379307 3243 tablet_bootstrap.cc:492] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: Bootstrap starting.
I20240503 13:59:00.380735 3244 tablet_bootstrap.cc:492] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: Bootstrap starting.
I20240503 13:59:00.380964 3245 tablet_bootstrap.cc:492] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: Bootstrap starting.
I20240503 13:59:00.382174 3243 tablet_bootstrap.cc:654] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:00.382855 3244 tablet_bootstrap.cc:654] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:00.383073 3243 log.cc:824] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:00.383122 3245 tablet_bootstrap.cc:654] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:00.383529 3244 log.cc:824] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:00.383807 3245 log.cc:824] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:00.384958 3243 tablet_bootstrap.cc:492] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: No bootstrap required, opened a new log
I20240503 13:59:00.385159 3243 ts_tablet_manager.cc:1367] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 13:59:00.385478 3245 tablet_bootstrap.cc:492] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: No bootstrap required, opened a new log
I20240503 13:59:00.385648 3245 ts_tablet_manager.cc:1367] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:00.387380 3244 tablet_bootstrap.cc:492] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: No bootstrap required, opened a new log
I20240503 13:59:00.387583 3244 ts_tablet_manager.cc:1367] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 13:59:00.389165 3243 raft_consensus.cc:348] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.389236 3245 raft_consensus.cc:348] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.389393 3245 raft_consensus.cc:374] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:00.389394 3243 raft_consensus.cc:374] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:00.389438 3245 raft_consensus.cc:729] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3746179b951d4ce1b3e5680a250eadeb, State: Initialized, Role: FOLLOWER
I20240503 13:59:00.389444 3243 raft_consensus.cc:729] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5682dac660e4f1c9d6b3b14f5315e01, State: Initialized, Role: FOLLOWER
I20240503 13:59:00.389920 3245 consensus_queue.cc:260] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.389921 3243 consensus_queue.cc:260] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.390866 3243 ts_tablet_manager.cc:1398] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:59:00.390866 3245 ts_tablet_manager.cc:1398] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:59:00.390887 3244 raft_consensus.cc:348] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.391100 3244 raft_consensus.cc:374] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:00.391198 3244 raft_consensus.cc:729] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed90aa0238274b3195b48db10fde8333, State: Initialized, Role: FOLLOWER
I20240503 13:59:00.391271 2701 heartbeater.cc:503] Master 127.0.99.190:34837 was elected leader, sending a full tablet report...
I20240503 13:59:00.391649 3244 consensus_queue.cc:260] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.392313 3094 heartbeater.cc:503] Master 127.0.99.190:34837 was elected leader, sending a full tablet report...
I20240503 13:59:00.392328 3225 heartbeater.cc:503] Master 127.0.99.190:34837 was elected leader, sending a full tablet report...
I20240503 13:59:00.392493 3244 ts_tablet_manager.cc:1398] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:00.496169 3251 raft_consensus.cc:482] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:00.496399 3251 raft_consensus.cc:504] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.497565 3251 leader_election.cc:288] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3746179b951d4ce1b3e5680a250eadeb (127.0.99.129:40247), b5682dac660e4f1c9d6b3b14f5315e01 (127.0.99.133:34453)
I20240503 13:59:00.501540 3179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f974c41e8497490ee4662d5f6364a" candidate_uuid: "ed90aa0238274b3195b48db10fde8333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" is_pre_election: true
I20240503 13:59:00.501596 2655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f974c41e8497490ee4662d5f6364a" candidate_uuid: "ed90aa0238274b3195b48db10fde8333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3746179b951d4ce1b3e5680a250eadeb" is_pre_election: true
I20240503 13:59:00.501871 2655 raft_consensus.cc:2454] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed90aa0238274b3195b48db10fde8333 in term 0.
I20240503 13:59:00.501870 3179 raft_consensus.cc:2454] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed90aa0238274b3195b48db10fde8333 in term 0.
I20240503 13:59:00.502391 2982 leader_election.cc:302] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3746179b951d4ce1b3e5680a250eadeb, ed90aa0238274b3195b48db10fde8333; no voters:
I20240503 13:59:00.502701 3251 raft_consensus.cc:2781] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:00.502779 3251 raft_consensus.cc:482] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:00.502810 3251 raft_consensus.cc:3037] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:00.503726 3251 raft_consensus.cc:504] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.504117 3251 leader_election.cc:288] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [CANDIDATE]: Term 1 election: Requested vote from peers 3746179b951d4ce1b3e5680a250eadeb (127.0.99.129:40247), b5682dac660e4f1c9d6b3b14f5315e01 (127.0.99.133:34453)
I20240503 13:59:00.504477 3179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f974c41e8497490ee4662d5f6364a" candidate_uuid: "ed90aa0238274b3195b48db10fde8333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5682dac660e4f1c9d6b3b14f5315e01"
I20240503 13:59:00.504503 2655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f974c41e8497490ee4662d5f6364a" candidate_uuid: "ed90aa0238274b3195b48db10fde8333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3746179b951d4ce1b3e5680a250eadeb"
I20240503 13:59:00.504591 3179 raft_consensus.cc:3037] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:00.504624 2655 raft_consensus.cc:3037] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:00.505574 2655 raft_consensus.cc:2454] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed90aa0238274b3195b48db10fde8333 in term 1.
I20240503 13:59:00.505580 3179 raft_consensus.cc:2454] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed90aa0238274b3195b48db10fde8333 in term 1.
I20240503 13:59:00.505920 2984 leader_election.cc:302] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5682dac660e4f1c9d6b3b14f5315e01, ed90aa0238274b3195b48db10fde8333; no voters:
I20240503 13:59:00.506131 3251 raft_consensus.cc:2781] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:00.506393 3251 raft_consensus.cc:686] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [term 1 LEADER]: Becoming Leader. State: Replica: ed90aa0238274b3195b48db10fde8333, State: Running, Role: LEADER
I20240503 13:59:00.506731 3251 consensus_queue.cc:237] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } }
I20240503 13:59:00.509927 2511 catalog_manager.cc:5484] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed90aa0238274b3195b48db10fde8333 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "ed90aa0238274b3195b48db10fde8333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed90aa0238274b3195b48db10fde8333" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:00.520184 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:59:00.521687 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3746179b951d4ce1b3e5680a250eadeb to finish bootstrapping
I20240503 13:59:00.532382 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d81a247057194f2ea0771201bbe827ad to finish bootstrapping
I20240503 13:59:00.538401 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0e9434614026420c8fbda104a9dc89b1 to finish bootstrapping
I20240503 13:59:00.544351 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ed90aa0238274b3195b48db10fde8333 to finish bootstrapping
W20240503 13:59:00.544955 2702 tablet.cc:2369] T 3d1f974c41e8497490ee4662d5f6364a P 3746179b951d4ce1b3e5680a250eadeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:00.549959 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b5682dac660e4f1c9d6b3b14f5315e01 to finish bootstrapping
W20240503 13:59:00.575428 3226 tablet.cc:2369] T 3d1f974c41e8497490ee4662d5f6364a P b5682dac660e4f1c9d6b3b14f5315e01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:00.638749 3095 tablet.cc:2369] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:00.749058 2832 heartbeater.cc:503] Master 127.0.99.190:34837 was elected leader, sending a full tablet report...
I20240503 13:59:00.752111 2897 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.752529 2635 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.752941 3159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.754024 3028 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:00.754107 2761 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.851004 3159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.851186 2761 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.851750 2635 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.852010 2897 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:00.862425 3028 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:00.890969 3253 consensus_queue.cc:1035] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3746179b951d4ce1b3e5680a250eadeb" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 40247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:00.897159 3327 consensus_queue.cc:1035] T 3d1f974c41e8497490ee4662d5f6364a P ed90aa0238274b3195b48db10fde8333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5682dac660e4f1c9d6b3b14f5315e01" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:00.956298 2963 heartbeater.cc:503] Master 127.0.99.190:34837 was elected leader, sending a full tablet report...
I20240503 13:59:01.088780 3028 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:01.089593 2897 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.091137 2761 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.098879 2635 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.100219 3159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.190577 2897 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.191483 3159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.192775 2635 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.193598 3028 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:01.194242 2761 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.301363 3159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.305961 2635 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.306494 3028 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:01.309486 2761 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.309557 2897 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:01.352356 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 2573
I20240503 13:59:01.357936 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 2704
I20240503 13:59:01.364812 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 2835
I20240503 13:59:01.370306 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 2966
I20240503 13:59:01.375627 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 3097
I20240503 13:59:01.381001 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 2482
2024-05-03T13:59:01Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2301 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2301 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240503 13:59:01.402642 398 test_util.cc:268] Using random seed: 523311115
I20240503 13:59:01.403451 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:01.403498 398 ts_itest-base.cc:113] --------------
I20240503 13:59:01.403512 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 13:59:01.403525 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 13:59:01.403538 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:01Z Disabled control of system clock
I20240503 13:59:01.411583 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43017
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43017
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 13:59:01.522781 3435 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 13:59:01.523084 3435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:01.523123 3435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:01.523196 3435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:01.526437 3435 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:01.526530 3435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:01.526563 3435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:01.526587 3435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:01.526618 3435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:01.530625 3435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43017
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:01.531636 3435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:01.532848 3435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:01.540905 3440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:01.541255 3441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:01.541818 3443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:01.542706 3435 server_base.cc:1034] running on GCE node
I20240503 13:59:01.543289 3435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:01.544333 3435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:01.545491 3435 hybrid_clock.cc:648] HybridClock initialized: now 1714744741545471 us; error 41 us; skew 500 ppm
I20240503 13:59:01.547585 3435 webserver.cc:460] Webserver started at http://127.0.99.190:38985/ using document root <none> and password file <none>
I20240503 13:59:01.548185 3435 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:01.548244 3435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:01.548439 3435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:01.550268 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5956e2ef385b4f099bcf90b9d7156148"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:01.550802 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5956e2ef385b4f099bcf90b9d7156148"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:01.554386 3435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 13:59:01.556805 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:01.557924 3435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:01.558091 3435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5956e2ef385b4f099bcf90b9d7156148"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:01.558197 3435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:01.580952 3435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:01.581709 3435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:01.581895 3435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:01.588958 3435 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43017
I20240503 13:59:01.588965 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43017 every 8 connection(s)
I20240503 13:59:01.590060 3435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:01.592715 3502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:01.596983 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 3435
I20240503 13:59:01.597108 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:01.599843 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148: Bootstrap starting.
I20240503 13:59:01.602109 3502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:01.602890 3502 log.cc:824] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:01.604858 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148: No bootstrap required, opened a new log
I20240503 13:59:01.607643 3502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5956e2ef385b4f099bcf90b9d7156148" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43017 } }
I20240503 13:59:01.607856 3502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:01.607903 3502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5956e2ef385b4f099bcf90b9d7156148, State: Initialized, Role: FOLLOWER
I20240503 13:59:01.608328 3502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5956e2ef385b4f099bcf90b9d7156148" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43017 } }
I20240503 13:59:01.608446 3502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:01.608505 3502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:01.608603 3502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:01.609402 3502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5956e2ef385b4f099bcf90b9d7156148" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43017 } }
I20240503 13:59:01.609752 3502 leader_election.cc:302] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5956e2ef385b4f099bcf90b9d7156148; no voters:
I20240503 13:59:01.610083 3502 leader_election.cc:288] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:01.610169 3507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:01.610374 3507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [term 1 LEADER]: Becoming Leader. State: Replica: 5956e2ef385b4f099bcf90b9d7156148, State: Running, Role: LEADER
I20240503 13:59:01.611025 3507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5956e2ef385b4f099bcf90b9d7156148" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43017 } }
I20240503 13:59:01.611243 3502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:01.612504 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5956e2ef385b4f099bcf90b9d7156148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5956e2ef385b4f099bcf90b9d7156148" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43017 } } }
I20240503 13:59:01.612608 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:01.613020 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:01.613049 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5956e2ef385b4f099bcf90b9d7156148. Latest consensus state: current_term: 1 leader_uuid: "5956e2ef385b4f099bcf90b9d7156148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5956e2ef385b4f099bcf90b9d7156148" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43017 } } }
I20240503 13:59:01.613121 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:01.616170 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:01.621886 3516 catalog_manager.cc:1347] Generated new cluster ID: 0d929ae800594eee9567fe2ee034ffd2
I20240503 13:59:01.622012 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:01.641762 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:01.643198 3516 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:01.658407 3516 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5956e2ef385b4f099bcf90b9d7156148: Generated new TSK 0
I20240503 13:59:01.659396 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:01.663630 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:01.781177 3526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:01.781430 3526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:01.781478 3526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:01.784953 3526 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:01.785038 3526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:01.785115 3526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:01.789090 3526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.3526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:01.790156 3526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:01.791405 3526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:01.800330 3531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:01.800784 3532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:01.801599 3534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:01.802426 3526 server_base.cc:1034] running on GCE node
I20240503 13:59:01.802878 3526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:01.803560 3526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:01.804733 3526 hybrid_clock.cc:648] HybridClock initialized: now 1714744741804703 us; error 39 us; skew 500 ppm
I20240503 13:59:01.806880 3526 webserver.cc:460] Webserver started at http://127.0.99.129:41293/ using document root <none> and password file <none>
I20240503 13:59:01.807504 3526 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:01.807564 3526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:01.807772 3526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:01.809659 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bc41cca530f54259aca15e34563208dc"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:01.810210 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bc41cca530f54259aca15e34563208dc"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:01.813920 3526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 13:59:01.816354 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:01.817484 3526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:01.817605 3526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bc41cca530f54259aca15e34563208dc"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:01.817698 3526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:01.841759 3526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:01.842538 3526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:01.842722 3526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:01.843305 3526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:01.844276 3526 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:01.844326 3526 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:01.844383 3526 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:01.844410 3526 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:01.852769 3526 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43281
I20240503 13:59:01.852787 3653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43281 every 8 connection(s)
I20240503 13:59:01.853796 3526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:01.860114 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 3526
I20240503 13:59:01.860229 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:01.862700 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:01.863052 3654 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:01.863332 3654 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:01.864059 3654 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:01.865741 3464 ts_manager.cc:194] Registered new tserver with Master: bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:01.867046 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:51791
W20240503 13:59:01.977207 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:01.977476 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:01.977523 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:01.980849 3657 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:01.980947 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:01.981026 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:01.984992 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.3657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:01.986092 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:01.987370 3657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:01.996275 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:01.997253 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:01.998097 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:01.998656 3657 server_base.cc:1034] running on GCE node
I20240503 13:59:01.999116 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:01.999773 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.000932 3657 hybrid_clock.cc:648] HybridClock initialized: now 1714744742000910 us; error 41 us; skew 500 ppm
I20240503 13:59:02.003110 3657 webserver.cc:460] Webserver started at http://127.0.99.130:44431/ using document root <none> and password file <none>
I20240503 13:59:02.003715 3657 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.003775 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.003974 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.005777 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3812cd29374a4b3485883810d0774867"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:02.006322 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3812cd29374a4b3485883810d0774867"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:02.009835 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:02.012229 3671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.013370 3657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:02.013522 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3812cd29374a4b3485883810d0774867"
format_stamp: "Formatted at 2024-05-03 13:59:01 on dist-test-slave-6fwv"
I20240503 13:59:02.013615 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.043183 3657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.043942 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.044116 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.044701 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.045675 3657 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.045727 3657 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.045784 3657 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.045809 3657 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.054514 3657 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:42107
I20240503 13:59:02.054545 3784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:42107 every 8 connection(s)
I20240503 13:59:02.055509 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:02.059520 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 3657
I20240503 13:59:02.059639 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:02.061663 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:02.064668 3785 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:02.064903 3785 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.065523 3785 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:02.066619 3464 ts_manager.cc:194] Registered new tserver with Master: 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:02.067200 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:36499
W20240503 13:59:02.174993 3788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.175256 3788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.175323 3788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.178579 3788 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.178669 3788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.178769 3788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:02.182693 3788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.3788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.183740 3788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.184980 3788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.194096 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.194486 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.195308 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.196013 3788 server_base.cc:1034] running on GCE node
I20240503 13:59:02.196456 3788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.197072 3788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.198238 3788 hybrid_clock.cc:648] HybridClock initialized: now 1714744742198203 us; error 52 us; skew 500 ppm
I20240503 13:59:02.200302 3788 webserver.cc:460] Webserver started at http://127.0.99.131:42377/ using document root <none> and password file <none>
I20240503 13:59:02.200877 3788 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.200935 3788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.201112 3788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.202972 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.203478 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.207054 3788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 13:59:02.209493 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.210659 3788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:02.210788 3788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.210880 3788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.249342 3788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.250108 3788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.250268 3788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.250834 3788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.251798 3788 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.251850 3788 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.251894 3788 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.251914 3788 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.260394 3788 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41619
I20240503 13:59:02.260416 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41619 every 8 connection(s)
I20240503 13:59:02.261431 3788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:02.268431 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 3788
I20240503 13:59:02.268530 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:02.270649 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:02.270644 3916 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:02.270891 3916 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.271483 3916 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:02.272542 3464 ts_manager.cc:194] Registered new tserver with Master: 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:02.273247 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49277
W20240503 13:59:02.383955 3919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.384222 3919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.384267 3919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.387595 3919 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.387682 3919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.387763 3919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:02.391718 3919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.3919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.392683 3919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.393898 3919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.403694 3924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.404217 3927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.406028 3925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.406002 3919 server_base.cc:1034] running on GCE node
I20240503 13:59:02.406487 3919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.407142 3919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.408310 3919 hybrid_clock.cc:648] HybridClock initialized: now 1714744742408292 us; error 58 us; skew 500 ppm
I20240503 13:59:02.410499 3919 webserver.cc:460] Webserver started at http://127.0.99.132:43571/ using document root <none> and password file <none>
I20240503 13:59:02.411135 3919 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.411198 3919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.411406 3919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.413357 3919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4e553631c0564b078cefd1cd1cdcc90d"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.413877 3919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4e553631c0564b078cefd1cd1cdcc90d"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.417562 3919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 13:59:02.420094 3933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.421299 3919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:02.421422 3919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4e553631c0564b078cefd1cd1cdcc90d"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.421533 3919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.435272 3919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.436023 3919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.436200 3919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.436797 3919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.437762 3919 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.437812 3919 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.437853 3919 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.437867 3919 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.446480 4046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36697 every 8 connection(s)
I20240503 13:59:02.446455 3919 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36697
I20240503 13:59:02.447607 3919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:02.456571 4047 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:02.456841 4047 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.457034 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 3919
I20240503 13:59:02.457134 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:02.457468 4047 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:02.458664 3464 ts_manager.cc:194] Registered new tserver with Master: 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697)
I20240503 13:59:02.459414 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55003
I20240503 13:59:02.459434 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:02.572165 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.572427 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.572472 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.575672 4050 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.575757 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.575829 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:02.579715 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.4050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.580708 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.581907 4050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.590854 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.591292 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.592165 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.592943 4050 server_base.cc:1034] running on GCE node
I20240503 13:59:02.593384 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.594020 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.595240 4050 hybrid_clock.cc:648] HybridClock initialized: now 1714744742595194 us; error 50 us; skew 500 ppm
I20240503 13:59:02.598366 4050 webserver.cc:460] Webserver started at http://127.0.99.133:39521/ using document root <none> and password file <none>
I20240503 13:59:02.598963 4050 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.599023 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.599215 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.601017 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.601526 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.605239 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:59:02.607658 4064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.608801 4050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:02.608919 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.609017 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.620141 4050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.620934 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.621104 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.621701 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.622781 4050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.622900 4050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.623018 4050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.623054 4050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.632592 4050 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39785
I20240503 13:59:02.632615 4177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39785 every 8 connection(s)
I20240503 13:59:02.633689 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:02.635478 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 4050
I20240503 13:59:02.635594 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:02.638258 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:02.643949 4178 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:02.644253 4178 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.645049 4178 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:02.646353 3464 ts_manager.cc:194] Registered new tserver with Master: 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785)
I20240503 13:59:02.647003 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:33751
W20240503 13:59:02.751039 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.751308 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.751353 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.754616 4181 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.754696 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.754781 4181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:02.758728 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.4181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.759765 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.760962 4181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.770236 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.770596 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.771473 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.772256 4181 server_base.cc:1034] running on GCE node
I20240503 13:59:02.772711 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.773370 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.774525 4181 hybrid_clock.cc:648] HybridClock initialized: now 1714744742774508 us; error 60 us; skew 500 ppm
I20240503 13:59:02.776621 4181 webserver.cc:460] Webserver started at http://127.0.99.134:40553/ using document root <none> and password file <none>
I20240503 13:59:02.777207 4181 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.777276 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.777472 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.779312 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.779805 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.783329 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:02.785713 4195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.786957 4181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:02.787083 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.787192 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.800674 4181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.801450 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.801618 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.802275 4181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.803265 4181 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.803315 4181 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.803370 4181 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.803392 4181 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.812445 4181 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43797
I20240503 13:59:02.812484 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43797 every 8 connection(s)
I20240503 13:59:02.813508 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:02.814328 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 4181
I20240503 13:59:02.814442 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:59:02.816897 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:02.823488 4309 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:02.823755 4309 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:02.824373 4309 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:02.825515 3464 ts_manager.cc:194] Registered new tserver with Master: 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797)
I20240503 13:59:02.826151 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:42497
I20240503 13:59:02.869621 3654 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
W20240503 13:59:02.934034 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:02.934306 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:02.934356 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:02.937620 4312 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:02.937703 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:02.937774 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:02.941795 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:02.942855 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:02.944092 4312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:02.953456 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.954142 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:02.954907 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:02.955551 4312 server_base.cc:1034] running on GCE node
I20240503 13:59:02.955997 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:02.956641 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:02.957813 4312 hybrid_clock.cc:648] HybridClock initialized: now 1714744742957788 us; error 45 us; skew 500 ppm
I20240503 13:59:02.960067 4312 webserver.cc:460] Webserver started at http://127.0.99.135:39383/ using document root <none> and password file <none>
I20240503 13:59:02.960691 4312 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:02.960763 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:02.960971 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:02.962987 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "07bdad76c7bd40afb3147833b3305924"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.963514 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "07bdad76c7bd40afb3147833b3305924"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.967420 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:02.970093 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.971297 4312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:02.971419 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "07bdad76c7bd40afb3147833b3305924"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:02.971511 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:02.987751 4312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:02.988526 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:02.988691 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:02.989277 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:02.990279 4312 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:02.990329 4312 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.990378 4312 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:02.990408 4312 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:02.999148 4312 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43643
I20240503 13:59:02.999167 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43643 every 8 connection(s)
I20240503 13:59:03.000214 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:03.003088 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 4312
I20240503 13:59:03.003217 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:59:03.009824 4440 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:03.010116 4440 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:03.010722 4440 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:03.011755 3464 ts_manager.cc:194] Registered new tserver with Master: 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643)
I20240503 13:59:03.012290 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:55061
I20240503 13:59:03.017731 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:59:03.027439 3464 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59536:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:59:03.046068 4243 tablet_service.cc:1467] Processing CreateTablet for tablet 8df17b9134914ca3a2c04800435b3697 (DEFAULT_TABLE table=TestTable [id=69946089d0414709a39ce318b85d074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:03.046068 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 8df17b9134914ca3a2c04800435b3697 (DEFAULT_TABLE table=TestTable [id=69946089d0414709a39ce318b85d074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:03.047058 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df17b9134914ca3a2c04800435b3697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:03.047084 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df17b9134914ca3a2c04800435b3697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:03.052228 4456 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:03.052248 4457 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap starting.
I20240503 13:59:03.054350 4456 tablet_bootstrap.cc:654] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:03.054559 4457 tablet_bootstrap.cc:654] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:03.055179 4456 log.cc:824] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:03.055217 4457 log.cc:824] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:03.056926 4457 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: No bootstrap required, opened a new log
I20240503 13:59:03.057114 4457 ts_tablet_manager.cc:1367] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:03.057114 4456 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:03.057277 4456 ts_tablet_manager.cc:1367] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:03.059994 4457 raft_consensus.cc:348] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.059983 4456 raft_consensus.cc:348] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.060174 4456 raft_consensus.cc:374] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:03.060184 4457 raft_consensus.cc:374] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:03.060212 4456 raft_consensus.cc:729] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:03.060231 4457 raft_consensus.cc:729] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Initialized, Role: FOLLOWER
I20240503 13:59:03.060662 4457 consensus_queue.cc:260] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.060662 4456 consensus_queue.cc:260] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.061358 4309 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
I20240503 13:59:03.061609 4457 ts_tablet_manager.cc:1398] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 13:59:03.061801 4456 ts_tablet_manager.cc:1398] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 13:59:03.062162 3916 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
W20240503 13:59:03.065336 4310 tablet.cc:2369] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:03.069247 3785 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
I20240503 13:59:03.179319 4461 raft_consensus.cc:482] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:03.179553 4461 raft_consensus.cc:504] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.180665 4461 leader_election.cc:288] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797)
I20240503 13:59:03.184700 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df17b9134914ca3a2c04800435b3697" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" is_pre_election: true
I20240503 13:59:03.185031 4263 raft_consensus.cc:2454] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:03.185524 3805 leader_election.cc:302] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 6d1a97f876fc4f8698f737fcf1b782ad; no voters:
I20240503 13:59:03.185818 4461 raft_consensus.cc:2781] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:03.185914 4461 raft_consensus.cc:482] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:03.185946 4461 raft_consensus.cc:3037] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:03.186919 4461 raft_consensus.cc:504] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.187253 4461 leader_election.cc:288] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797)
I20240503 13:59:03.187784 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df17b9134914ca3a2c04800435b3697" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
I20240503 13:59:03.187922 4263 raft_consensus.cc:3037] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:03.188860 4263 raft_consensus.cc:2454] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:03.189196 3805 leader_election.cc:302] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 6d1a97f876fc4f8698f737fcf1b782ad; no voters:
I20240503 13:59:03.189378 4461 raft_consensus.cc:2781] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:03.189613 4461 raft_consensus.cc:686] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:03.189953 4461 consensus_queue.cc:237] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:03.193228 3466 catalog_manager.cc:5484] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:03.235787 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:59:03.237206 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bc41cca530f54259aca15e34563208dc to finish bootstrapping
I20240503 13:59:03.243394 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3812cd29374a4b3485883810d0774867 to finish bootstrapping
I20240503 13:59:03.249267 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0c6ca23f9d9b4c91a6347c0924674bdd to finish bootstrapping
I20240503 13:59:03.254434 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4e553631c0564b078cefd1cd1cdcc90d to finish bootstrapping
I20240503 13:59:03.260470 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 14a79ecc8c2c45adac20a5239cd45fd2 to finish bootstrapping
W20240503 13:59:03.262916 3917 tablet.cc:2369] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:03.266805 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6d1a97f876fc4f8698f737fcf1b782ad to finish bootstrapping
I20240503 13:59:03.272370 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 07bdad76c7bd40afb3147833b3305924 to finish bootstrapping
I20240503 13:59:03.278524 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 3919
I20240503 13:59:03.283648 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 4050
I20240503 13:59:03.288902 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 4181
I20240503 13:59:03.294095 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 4312
W20240503 13:59:03.653817 3805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43797: connect: Connection refused (error 111)
W20240503 13:59:03.654750 3805 consensus_peers.cc:485] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43797: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:05.985615 3805 consensus_peers.cc:485] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43797: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:59:07.050303 3466 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59536:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:07.065289 3586 tablet_service.cc:1467] Processing CreateTablet for tablet c1c58189e2df41c3b8b4a0924ae42a03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.065289 3717 tablet_service.cc:1467] Processing CreateTablet for tablet c1c58189e2df41c3b8b4a0924ae42a03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.065301 3588 tablet_service.cc:1467] Processing CreateTablet for tablet bd23523f46044bb7b7e4f44056687c43 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.066440 3586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c58189e2df41c3b8b4a0924ae42a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.066440 3717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c58189e2df41c3b8b4a0924ae42a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.066841 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd23523f46044bb7b7e4f44056687c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.069392 3587 tablet_service.cc:1467] Processing CreateTablet for tablet 97d59e7bd0da44d9a7af791f2477b183 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.069693 3587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d59e7bd0da44d9a7af791f2477b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.070148 3719 tablet_service.cc:1467] Processing CreateTablet for tablet bd23523f46044bb7b7e4f44056687c43 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.070431 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd23523f46044bb7b7e4f44056687c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.073611 4485 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:07.065310 3718 tablet_service.cc:1467] Processing CreateTablet for tablet 97d59e7bd0da44d9a7af791f2477b183 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.074209 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d59e7bd0da44d9a7af791f2477b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.074189 4486 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:07.076956 4485 tablet_bootstrap.cc:654] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.077579 4486 tablet_bootstrap.cc:654] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.078162 4485 log.cc:824] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:07.078534 4486 log.cc:824] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:07.079857 4485 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:07.080049 4485 ts_tablet_manager.cc:1367] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 13:59:07.080186 4486 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:07.080380 4486 ts_tablet_manager.cc:1367] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 13:59:07.082888 4485 raft_consensus.cc:348] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.083108 4485 raft_consensus.cc:374] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.083160 4485 raft_consensus.cc:729] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.083129 4486 raft_consensus.cc:348] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.083330 4486 raft_consensus.cc:374] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.083436 4486 raft_consensus.cc:729] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.083627 4485 consensus_queue.cc:260] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.083799 4486 consensus_queue.cc:260] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.084550 4485 ts_tablet_manager.cc:1398] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:07.084913 4485 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:07.085531 4486 ts_tablet_manager.cc:1398] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:07.085763 4486 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:07.085963 4485 tablet_bootstrap.cc:654] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.086707 4486 tablet_bootstrap.cc:654] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.087239 4485 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:07.087333 4485 ts_tablet_manager.cc:1367] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:07.087975 4485 raft_consensus.cc:348] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.088086 4485 raft_consensus.cc:374] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.088117 4485 raft_consensus.cc:729] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.088261 4485 consensus_queue.cc:260] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.088496 4485 ts_tablet_manager.cc:1398] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:07.089257 4486 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:07.089381 4485 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:07.089396 4486 ts_tablet_manager.cc:1367] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:07.089967 4486 raft_consensus.cc:348] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.090142 4486 raft_consensus.cc:374] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.090190 4486 raft_consensus.cc:729] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.090354 4485 tablet_bootstrap.cc:654] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.090346 4486 consensus_queue.cc:260] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.090591 4486 ts_tablet_manager.cc:1398] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.090742 4486 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:07.091650 4486 tablet_bootstrap.cc:654] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.091660 4485 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:07.091737 4485 ts_tablet_manager.cc:1367] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:07.092144 4485 raft_consensus.cc:348] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.092262 4485 raft_consensus.cc:374] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.092305 4485 raft_consensus.cc:729] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.092437 4485 consensus_queue.cc:260] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.092702 4485 ts_tablet_manager.cc:1398] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:07.092777 4486 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:07.092862 4486 ts_tablet_manager.cc:1367] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:07.093413 4486 raft_consensus.cc:348] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.093519 4486 raft_consensus.cc:374] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.093547 4486 raft_consensus.cc:729] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.093739 4486 consensus_queue.cc:260] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.094069 4486 ts_tablet_manager.cc:1398] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20240503 13:59:07.109030 3655 tablet.cc:2369] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:07.158077 4490 raft_consensus.cc:482] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.158282 4490 raft_consensus.cc:504] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.159381 4490 leader_election.cc:288] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:07.163724 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183" candidate_uuid: "bc41cca530f54259aca15e34563208dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867" is_pre_election: true
I20240503 13:59:07.164399 3739 raft_consensus.cc:2454] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc41cca530f54259aca15e34563208dc in term 0.
I20240503 13:59:07.164870 3541 leader_election.cc:302] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.165134 4490 raft_consensus.cc:2781] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.165239 4490 raft_consensus.cc:482] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.165270 4490 raft_consensus.cc:3037] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.166239 4490 raft_consensus.cc:504] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.166577 4490 leader_election.cc:288] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:07.166998 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183" candidate_uuid: "bc41cca530f54259aca15e34563208dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:07.167135 3739 raft_consensus.cc:3037] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.168046 3739 raft_consensus.cc:2454] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc41cca530f54259aca15e34563208dc in term 1.
I20240503 13:59:07.168354 3541 leader_election.cc:302] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.168531 4490 raft_consensus.cc:2781] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.168747 4490 raft_consensus.cc:686] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Becoming Leader. State: Replica: bc41cca530f54259aca15e34563208dc, State: Running, Role: LEADER
I20240503 13:59:07.169065 4490 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.172391 3464 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc reported cstate change: term changed from 0 to 1, leader changed from <none> to bc41cca530f54259aca15e34563208dc (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:07.210384 4491 raft_consensus.cc:482] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.210525 4491 raft_consensus.cc:504] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.211570 4491 leader_election.cc:288] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:07.215509 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c58189e2df41c3b8b4a0924ae42a03" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:07.215749 3608 raft_consensus.cc:2454] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:07.216173 3673 leader_election.cc:302] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.216462 4491 raft_consensus.cc:2781] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.216542 4491 raft_consensus.cc:482] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.216578 4491 raft_consensus.cc:3037] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.217486 4491 raft_consensus.cc:504] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.217809 4491 leader_election.cc:288] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:07.218257 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c58189e2df41c3b8b4a0924ae42a03" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:07.218406 3608 raft_consensus.cc:3037] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.219328 3608 raft_consensus.cc:2454] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:07.219625 3673 leader_election.cc:302] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.219799 4491 raft_consensus.cc:2781] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.220028 4491 raft_consensus.cc:686] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:07.220399 4491 consensus_queue.cc:237] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.223666 3464 catalog_manager.cc:5484] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:07.310570 3786 tablet.cc:2369] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:07.364420 4491 raft_consensus.cc:482] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.364579 4491 raft_consensus.cc:504] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.364986 4491 leader_election.cc:288] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:07.365622 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd23523f46044bb7b7e4f44056687c43" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:07.365782 3608 raft_consensus.cc:2454] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:07.366135 3673 leader_election.cc:302] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.366338 4491 raft_consensus.cc:2781] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.366395 4491 raft_consensus.cc:482] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.366431 4491 raft_consensus.cc:3037] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.367295 4491 raft_consensus.cc:504] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.367630 4491 leader_election.cc:288] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:07.368144 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd23523f46044bb7b7e4f44056687c43" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:07.368265 3608 raft_consensus.cc:3037] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.369065 3608 raft_consensus.cc:2454] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:07.369422 3673 leader_election.cc:302] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.369611 4491 raft_consensus.cc:2781] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.369726 4491 raft_consensus.cc:686] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:07.369866 4491 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.371573 3464 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:07.531929 4503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:07.532197 4503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:07.535352 4503 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 13:59:07.632437 3608 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.632951 3607 raft_consensus.cc:1261] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.633414 3739 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:07.633728 4500 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.634212 4490 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.634251 4500 consensus_queue.cc:1035] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:07.642241 4533 mvcc.cc:205] Tried to move back new op lower bound from 7023594486297411584 to 7023594484406763520. Current Snapshot: MvccSnapshot[applied={T|T < 7023594486297411584}]
I20240503 13:59:07.642568 4532 mvcc.cc:205] Tried to move back new op lower bound from 7023594486293725184 to 7023594485228232704. Current Snapshot: MvccSnapshot[applied={T|T < 7023594486293725184}]
INSERT report
rows total: 2000
time total: 92.8826 ms
time per row: 0.0464413 ms
I20240503 13:59:07.679324 3460 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59536:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:07.685492 3586 tablet_service.cc:1467] Processing CreateTablet for tablet dcf42adb4ddc4a25907a5c5fa94e31b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.685446 3850 tablet_service.cc:1467] Processing CreateTablet for tablet dcf42adb4ddc4a25907a5c5fa94e31b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.685446 3849 tablet_service.cc:1467] Processing CreateTablet for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e (DEFAULT_TABLE table=rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.685663 3717 tablet_service.cc:1467] Processing CreateTablet for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e (DEFAULT_TABLE table=rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.685846 3586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf42adb4ddc4a25907a5c5fa94e31b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.685849 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf42adb4ddc4a25907a5c5fa94e31b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.685918 3717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.686069 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.688277 4553 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:07.689016 3586 tablet_service.cc:1467] Processing CreateTablet for tablet adfa836cb94c4050bdb65e007b232f2b (DEFAULT_TABLE table=rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.689141 4553 tablet_bootstrap.cc:654] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.689148 3850 tablet_service.cc:1467] Processing CreateTablet for tablet adfa836cb94c4050bdb65e007b232f2b (DEFAULT_TABLE table=rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:07.689270 3586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfa836cb94c4050bdb65e007b232f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.689399 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfa836cb94c4050bdb65e007b232f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.689769 4555 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:07.689852 4554 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:07.690688 4555 tablet_bootstrap.cc:654] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.690757 4554 tablet_bootstrap.cc:654] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.692065 4555 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:07.692140 4554 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:07.692167 4555 ts_tablet_manager.cc:1367] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:07.692165 4553 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:07.692207 4554 ts_tablet_manager.cc:1367] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:07.692252 4553 ts_tablet_manager.cc:1367] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:07.692670 4554 raft_consensus.cc:348] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.692749 4555 raft_consensus.cc:348] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.692795 4554 raft_consensus.cc:374] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.692835 4554 raft_consensus.cc:729] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.692755 4553 raft_consensus.cc:348] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.692838 4555 raft_consensus.cc:374] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.692871 4555 raft_consensus.cc:729] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.692878 4553 raft_consensus.cc:374] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.692909 4553 raft_consensus.cc:729] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.692996 4555 consensus_queue.cc:260] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.692999 4554 consensus_queue.cc:260] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.693058 4553 consensus_queue.cc:260] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.693271 4554 ts_tablet_manager.cc:1398] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.693272 4555 ts_tablet_manager.cc:1398] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.693322 4553 ts_tablet_manager.cc:1398] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.693490 4553 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:07.694001 4554 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:07.694494 4553 tablet_bootstrap.cc:654] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.695061 4554 tablet_bootstrap.cc:654] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.696336 4553 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:07.696548 4553 ts_tablet_manager.cc:1367] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:07.697038 4553 raft_consensus.cc:348] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.697167 4553 raft_consensus.cc:374] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.697204 4553 raft_consensus.cc:729] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.697350 4553 consensus_queue.cc:260] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.697470 4554 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:07.697558 4554 ts_tablet_manager.cc:1367] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:07.697602 4553 ts_tablet_manager.cc:1398] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.697780 4553 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:07.698015 4554 raft_consensus.cc:348] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.698112 4554 raft_consensus.cc:374] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.698148 4554 raft_consensus.cc:729] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.698256 4554 consensus_queue.cc:260] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.698516 4554 ts_tablet_manager.cc:1398] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:07.698995 4553 tablet_bootstrap.cc:654] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:07.700064 4553 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:07.700146 4553 ts_tablet_manager.cc:1367] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:07.700520 4553 raft_consensus.cc:348] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.700603 4553 raft_consensus.cc:374] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:07.700623 4553 raft_consensus.cc:729] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:07.700724 4553 consensus_queue.cc:260] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.700953 4553 ts_tablet_manager.cc:1398] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:07.729604 4490 raft_consensus.cc:482] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.729754 4490 raft_consensus.cc:504] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.730253 4490 leader_election.cc:288] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:07.733882 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfa836cb94c4050bdb65e007b232f2b" candidate_uuid: "bc41cca530f54259aca15e34563208dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" is_pre_election: true
I20240503 13:59:07.734149 3870 raft_consensus.cc:2454] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc41cca530f54259aca15e34563208dc in term 0.
I20240503 13:59:07.734498 3543 leader_election.cc:302] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.734695 4490 raft_consensus.cc:2781] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.734760 4490 raft_consensus.cc:482] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.734794 4490 raft_consensus.cc:3037] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.735637 4490 raft_consensus.cc:504] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.735951 4490 leader_election.cc:288] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 election: Requested vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:07.736351 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfa836cb94c4050bdb65e007b232f2b" candidate_uuid: "bc41cca530f54259aca15e34563208dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
I20240503 13:59:07.736464 3870 raft_consensus.cc:3037] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.737255 3870 raft_consensus.cc:2454] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc41cca530f54259aca15e34563208dc in term 1.
I20240503 13:59:07.737547 3543 leader_election.cc:302] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.737715 4490 raft_consensus.cc:2781] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.737802 4490 raft_consensus.cc:686] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Becoming Leader. State: Replica: bc41cca530f54259aca15e34563208dc, State: Running, Role: LEADER
I20240503 13:59:07.737944 4490 consensus_queue.cc:237] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.739651 3466 catalog_manager.cc:5484] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc reported cstate change: term changed from 0 to 1, leader changed from <none> to bc41cca530f54259aca15e34563208dc (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:07.859519 4535 raft_consensus.cc:482] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.859681 4535 raft_consensus.cc:504] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.860167 4535 leader_election.cc:288] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:07.864131 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf42adb4ddc4a25907a5c5fa94e31b7" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:07.864306 3608 raft_consensus.cc:2454] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:07.864670 3804 leader_election.cc:302] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.864889 4535 raft_consensus.cc:2781] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.864974 4535 raft_consensus.cc:482] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.865007 4535 raft_consensus.cc:3037] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.866160 4535 raft_consensus.cc:504] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.866528 4535 leader_election.cc:288] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:07.866991 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf42adb4ddc4a25907a5c5fa94e31b7" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:07.867121 3608 raft_consensus.cc:3037] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.867942 3608 raft_consensus.cc:2454] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:07.868268 3804 leader_election.cc:302] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:07.868435 4535 raft_consensus.cc:2781] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.868655 4535 raft_consensus.cc:686] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:07.868821 4535 consensus_queue.cc:237] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } }
I20240503 13:59:07.870570 3466 catalog_manager.cc:5484] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:07.981504 4544 raft_consensus.cc:482] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:07.981655 4544 raft_consensus.cc:504] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.982147 4544 leader_election.cc:288] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:07.985612 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ebb671e56f4e29adb6a2c0a7c6b00e" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" is_pre_election: true
I20240503 13:59:07.985785 3870 raft_consensus.cc:2454] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:07.986097 3674 leader_election.cc:302] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:07.986311 4544 raft_consensus.cc:2781] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:07.986377 4544 raft_consensus.cc:482] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:07.986404 4544 raft_consensus.cc:3037] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.987237 4544 raft_consensus.cc:504] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.987555 4544 leader_election.cc:288] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:07.987946 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ebb671e56f4e29adb6a2c0a7c6b00e" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
I20240503 13:59:07.988068 3870 raft_consensus.cc:3037] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:07.988819 3870 raft_consensus.cc:2454] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:07.989131 3674 leader_election.cc:302] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:07.989320 4544 raft_consensus.cc:2781] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:07.989413 4544 raft_consensus.cc:686] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:07.989555 4544 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:07.991284 3466 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:08.171763 4569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:08.172055 4569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:08.175387 4569 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:59:08.260200 3870 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.260836 4544 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.260232 3869 raft_consensus.cc:1261] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.266326 4490 consensus_queue.cc:1035] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.269783 3608 raft_consensus.cc:1261] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.270362 4535 consensus_queue.cc:1035] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.276049 4606 mvcc.cc:205] Tried to move back new op lower bound from 7023594488868634624 to 7023594487766102016. Current Snapshot: MvccSnapshot[applied={T|T < 7023594488868634624}]
INSERT report
rows total: 2000
time total: 93.383 ms
time per row: 0.0466915 ms
I20240503 13:59:08.313819 3466 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59536:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:08.319849 3850 tablet_service.cc:1467] Processing CreateTablet for tablet fc8c685a1dd4496b9be2b1e09d237371 (DEFAULT_TABLE table=rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.319840 3717 tablet_service.cc:1467] Processing CreateTablet for tablet fc8c685a1dd4496b9be2b1e09d237371 (DEFAULT_TABLE table=rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.320174 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8c685a1dd4496b9be2b1e09d237371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.320175 3717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8c685a1dd4496b9be2b1e09d237371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.320169 3718 tablet_service.cc:1467] Processing CreateTablet for tablet 480d84e5600d40e7b8c7fe001ba33c9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.320276 3586 tablet_service.cc:1467] Processing CreateTablet for tablet 480d84e5600d40e7b8c7fe001ba33c9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.320482 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480d84e5600d40e7b8c7fe001ba33c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.320515 3586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480d84e5600d40e7b8c7fe001ba33c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.322696 4625 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:08.322734 3849 tablet_service.cc:1467] Processing CreateTablet for tablet e572db34c8844384b728068168645ecc (DEFAULT_TABLE table=rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.322963 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572db34c8844384b728068168645ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.323333 4626 tablet_bootstrap.cc:492] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:08.323611 4625 tablet_bootstrap.cc:654] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.324235 4626 tablet_bootstrap.cc:654] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.324370 4624 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:08.325088 3719 tablet_service.cc:1467] Processing CreateTablet for tablet e572db34c8844384b728068168645ecc (DEFAULT_TABLE table=rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.325191 4624 tablet_bootstrap.cc:654] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.325332 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572db34c8844384b728068168645ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.325367 4626 tablet_bootstrap.cc:492] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:08.325448 4626 ts_tablet_manager.cc:1367] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:08.325901 4626 raft_consensus.cc:348] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.325968 4625 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:08.326048 4626 raft_consensus.cc:374] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.326082 4626 raft_consensus.cc:729] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.326115 4625 ts_tablet_manager.cc:1367] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:08.326242 4626 consensus_queue.cc:260] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.326604 4626 ts_tablet_manager.cc:1398] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.327055 4625 raft_consensus.cc:348] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.327168 4625 raft_consensus.cc:374] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.327204 4625 raft_consensus.cc:729] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.327328 4625 consensus_queue.cc:260] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.327622 4625 ts_tablet_manager.cc:1398] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:08.327800 4625 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:08.328730 4625 tablet_bootstrap.cc:654] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.329661 4624 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:08.329761 4624 ts_tablet_manager.cc:1367] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:08.330148 4624 raft_consensus.cc:348] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.330246 4624 raft_consensus.cc:374] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.330266 4624 raft_consensus.cc:729] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.330375 4624 consensus_queue.cc:260] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.330569 4624 ts_tablet_manager.cc:1398] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.330699 4624 tablet_bootstrap.cc:492] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:08.331501 4624 tablet_bootstrap.cc:654] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.332394 4625 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:08.332480 4625 ts_tablet_manager.cc:1367] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:08.332640 4624 tablet_bootstrap.cc:492] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:08.332739 4624 ts_tablet_manager.cc:1367] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:08.332924 4625 raft_consensus.cc:348] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.333024 4625 raft_consensus.cc:374] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.333053 4625 raft_consensus.cc:729] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.333163 4625 consensus_queue.cc:260] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.333446 4625 ts_tablet_manager.cc:1398] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:08.333738 4624 raft_consensus.cc:348] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.333848 4624 raft_consensus.cc:374] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.333882 4624 raft_consensus.cc:729] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.334014 4624 consensus_queue.cc:260] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.334265 4624 ts_tablet_manager.cc:1398] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.334426 4624 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:08.335248 4624 tablet_bootstrap.cc:654] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.336418 4624 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:08.336495 4624 ts_tablet_manager.cc:1367] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:08.336946 4624 raft_consensus.cc:348] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.337055 4624 raft_consensus.cc:374] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.337086 4624 raft_consensus.cc:729] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.337204 4624 consensus_queue.cc:260] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.337455 4624 ts_tablet_manager.cc:1398] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:08.361707 4622 raft_consensus.cc:482] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.361847 4622 raft_consensus.cc:504] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.362349 4622 leader_election.cc:288] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:08.365788 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8c685a1dd4496b9be2b1e09d237371" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867" is_pre_election: true
I20240503 13:59:08.365952 3739 raft_consensus.cc:2454] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:08.366297 3803 leader_election.cc:302] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:08.366485 4622 raft_consensus.cc:2781] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.366549 4622 raft_consensus.cc:482] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.366580 4622 raft_consensus.cc:3037] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.367444 4622 raft_consensus.cc:504] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.367739 4622 leader_election.cc:288] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:08.368134 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc8c685a1dd4496b9be2b1e09d237371" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:08.368249 3739 raft_consensus.cc:3037] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.369025 3739 raft_consensus.cc:2454] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:08.369330 3803 leader_election.cc:302] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:08.369511 4622 raft_consensus.cc:2781] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.369621 4622 raft_consensus.cc:686] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:08.369812 4622 consensus_queue.cc:237] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.371558 3466 catalog_manager.cc:5484] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:08.432035 4622 raft_consensus.cc:482] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.432193 4622 raft_consensus.cc:504] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.432570 4622 leader_election.cc:288] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:08.433113 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572db34c8844384b728068168645ecc" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867" is_pre_election: true
I20240503 13:59:08.433264 3739 raft_consensus.cc:2454] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:08.433562 3803 leader_election.cc:302] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:08.433743 4622 raft_consensus.cc:2781] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.433823 4622 raft_consensus.cc:482] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.433861 4622 raft_consensus.cc:3037] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.434746 4622 raft_consensus.cc:504] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.435057 4622 leader_election.cc:288] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:08.435477 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572db34c8844384b728068168645ecc" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:08.435621 3739 raft_consensus.cc:3037] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.436416 3739 raft_consensus.cc:2454] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:08.436719 3803 leader_election.cc:302] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:08.436913 4622 raft_consensus.cc:2781] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.437005 4622 raft_consensus.cc:686] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:08.437146 4622 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.438748 3466 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:08.521147 4544 raft_consensus.cc:482] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.521292 4544 raft_consensus.cc:504] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.521677 4544 leader_election.cc:288] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:08.522228 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d84e5600d40e7b8c7fe001ba33c9f" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:08.522377 3608 raft_consensus.cc:2454] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:08.522708 3673 leader_election.cc:302] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:08.522929 4544 raft_consensus.cc:2781] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.523001 4544 raft_consensus.cc:482] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.523046 4544 raft_consensus.cc:3037] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.523895 4544 raft_consensus.cc:504] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.524204 4544 leader_election.cc:288] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:08.524595 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d84e5600d40e7b8c7fe001ba33c9f" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:08.524717 3608 raft_consensus.cc:3037] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.525478 3608 raft_consensus.cc:2454] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:08.525822 3673 leader_election.cc:302] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:08.526039 4544 raft_consensus.cc:2781] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.526135 4544 raft_consensus.cc:686] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:08.526278 4544 consensus_queue.cc:237] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.527851 3466 catalog_manager.cc:5484] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:08.682575 4636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:08.682880 4636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:08.686184 4636 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:59:08.749729 4622 consensus_queue.cc:1035] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.780019 3608 raft_consensus.cc:1261] T 480d84e5600d40e7b8c7fe001ba33c9f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.780567 4544 consensus_queue.cc:1035] T 480d84e5600d40e7b8c7fe001ba33c9f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:08.780807 3738 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:08.781299 4622 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.7117 ms
time per row: 0.0488559 ms
I20240503 13:59:08.829371 3458 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59536:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:08.834920 3586 tablet_service.cc:1467] Processing CreateTablet for tablet 37a9c633962b40f1b42376cf597ad415 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.834959 3849 tablet_service.cc:1467] Processing CreateTablet for tablet 37a9c633962b40f1b42376cf597ad415 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.835263 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a9c633962b40f1b42376cf597ad415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.835263 3586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a9c633962b40f1b42376cf597ad415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.837080 4672 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:08.837088 4673 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:08.837800 3719 tablet_service.cc:1467] Processing CreateTablet for tablet e967e4257bda4e4fa9b3fdfadd22368f (DEFAULT_TABLE table=rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.838116 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e967e4257bda4e4fa9b3fdfadd22368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.838145 4673 tablet_bootstrap.cc:654] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.838188 4672 tablet_bootstrap.cc:654] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.839242 4673 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:08.839347 4673 ts_tablet_manager.cc:1367] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:08.839803 4673 raft_consensus.cc:348] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.839926 4673 raft_consensus.cc:374] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.839967 4673 raft_consensus.cc:729] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.840118 4673 consensus_queue.cc:260] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.840350 4676 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:08.840410 4673 ts_tablet_manager.cc:1398] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.840855 3586 tablet_service.cc:1467] Processing CreateTablet for tablet 809441335aa64f959ef084cb3c8b2434 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.840999 3587 tablet_service.cc:1467] Processing CreateTablet for tablet e967e4257bda4e4fa9b3fdfadd22368f (DEFAULT_TABLE table=rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.841113 3586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809441335aa64f959ef084cb3c8b2434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.841313 4676 tablet_bootstrap.cc:654] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.841483 3587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e967e4257bda4e4fa9b3fdfadd22368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.843329 3849 tablet_service.cc:1467] Processing CreateTablet for tablet 809441335aa64f959ef084cb3c8b2434 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:08.843566 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809441335aa64f959ef084cb3c8b2434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:08.845124 4673 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:08.845700 4672 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:08.845814 4672 ts_tablet_manager.cc:1367] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:08.845964 4676 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:08.846058 4673 tablet_bootstrap.cc:654] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.846087 4676 ts_tablet_manager.cc:1367] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:08.846341 4672 raft_consensus.cc:348] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.846549 4676 raft_consensus.cc:348] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.846452 4672 raft_consensus.cc:374] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.846652 4676 raft_consensus.cc:374] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.846657 4672 raft_consensus.cc:729] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.846683 4676 raft_consensus.cc:729] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.846839 4676 consensus_queue.cc:260] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.847046 4672 consensus_queue.cc:260] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.847133 4676 ts_tablet_manager.cc:1398] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.847437 4672 ts_tablet_manager.cc:1398] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:08.847592 4672 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:08.848093 4673 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:08.848171 4673 ts_tablet_manager.cc:1367] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:08.848644 4673 raft_consensus.cc:348] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.848903 4673 raft_consensus.cc:374] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.848942 4673 raft_consensus.cc:729] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.849021 4672 tablet_bootstrap.cc:654] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:08.849098 4673 consensus_queue.cc:260] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.849537 4673 ts_tablet_manager.cc:1398] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.851315 4672 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:08.851400 4672 ts_tablet_manager.cc:1367] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:08.851866 4672 raft_consensus.cc:348] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.851976 4672 raft_consensus.cc:374] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.852025 4672 raft_consensus.cc:729] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.852145 4672 consensus_queue.cc:260] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.852424 4672 ts_tablet_manager.cc:1398] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:08.852576 4672 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:08.853358 4672 tablet_bootstrap.cc:654] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
W20240503 13:59:08.854815 3805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43797: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 13:59:08.855626 3805 consensus_peers.cc:485] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43797: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:59:08.856384 4672 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:08.856469 4672 ts_tablet_manager.cc:1367] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:08.857117 4672 raft_consensus.cc:348] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.857223 4672 raft_consensus.cc:374] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:08.857251 4672 raft_consensus.cc:729] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:08.857379 4672 consensus_queue.cc:260] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.857645 4672 ts_tablet_manager.cc:1398] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:08.872519 4500 raft_consensus.cc:482] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.872655 4500 raft_consensus.cc:504] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.873001 4500 leader_election.cc:288] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:08.873431 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:08.873569 3608 raft_consensus.cc:2454] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:08.873844 3673 leader_election.cc:302] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:08.874042 4500 raft_consensus.cc:2781] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.874111 4500 raft_consensus.cc:482] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.874145 4500 raft_consensus.cc:3037] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.874909 4500 raft_consensus.cc:504] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.875180 4500 leader_election.cc:288] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:08.875586 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:08.875692 3608 raft_consensus.cc:3037] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.876412 3608 raft_consensus.cc:2454] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:08.876683 3673 leader_election.cc:302] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:08.876868 4500 raft_consensus.cc:2781] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.876960 4500 raft_consensus.cc:686] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:08.877100 4500 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:08.878716 3461 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:08.879853 4535 raft_consensus.cc:482] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:08.879961 4535 raft_consensus.cc:504] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.880275 4535 leader_election.cc:288] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:08.880805 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:08.880925 3608 raft_consensus.cc:2454] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:08.881310 3804 leader_election.cc:302] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:08.881489 4535 raft_consensus.cc:2781] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:08.881549 4535 raft_consensus.cc:482] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:08.881583 4535 raft_consensus.cc:3037] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.882413 4535 raft_consensus.cc:504] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.882673 4535 leader_election.cc:288] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:08.883067 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:08.883183 3608 raft_consensus.cc:3037] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:08.883908 3608 raft_consensus.cc:2454] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:08.884181 3804 leader_election.cc:302] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:08.884370 4535 raft_consensus.cc:2781] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:08.884449 4535 raft_consensus.cc:686] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:08.884552 4535 consensus_queue.cc:237] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:08.886109 3461 catalog_manager.cc:5484] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:09.032683 4535 raft_consensus.cc:482] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.032825 4535 raft_consensus.cc:504] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.033169 4535 leader_election.cc:288] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:09.033628 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a9c633962b40f1b42376cf597ad415" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:09.033773 3608 raft_consensus.cc:2454] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:09.034088 3804 leader_election.cc:302] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:09.034258 4535 raft_consensus.cc:2781] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.034320 4535 raft_consensus.cc:482] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.034353 4535 raft_consensus.cc:3037] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.035161 4535 raft_consensus.cc:504] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.035442 4535 leader_election.cc:288] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:09.035833 3608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a9c633962b40f1b42376cf597ad415" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:09.035956 3608 raft_consensus.cc:3037] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.036716 3608 raft_consensus.cc:2454] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:09.036996 3804 leader_election.cc:302] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:09.037147 4535 raft_consensus.cc:2781] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.037236 4535 raft_consensus.cc:686] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:09.037389 4535 consensus_queue.cc:237] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.039028 3458 catalog_manager.cc:5484] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:09.188086 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:09.188385 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:09.191632 4685 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:59:09.282929 3608 raft_consensus.cc:1261] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.283382 4565 consensus_queue.cc:1035] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.284548 3607 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.285135 4500 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.290299 3607 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.300446 4535 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.724 ms
time per row: 0.0513621 ms
I20240503 13:59:09.338121 3458 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59536:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:09.344339 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 49afc417528049839897ee5600b954e8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.344347 3719 tablet_service.cc:1467] Processing CreateTablet for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.344585 3587 tablet_service.cc:1467] Processing CreateTablet for tablet 2e7fa1300123438896b7be116d5560fa (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.344679 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.344679 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49afc417528049839897ee5600b954e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.344836 3587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e7fa1300123438896b7be116d5560fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.346767 4672 tablet_bootstrap.cc:492] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc: Bootstrap starting.
I20240503 13:59:09.346936 4673 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:09.347059 3718 tablet_service.cc:1467] Processing CreateTablet for tablet 2e7fa1300123438896b7be116d5560fa (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.347080 3849 tablet_service.cc:1467] Processing CreateTablet for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.347308 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.347323 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e7fa1300123438896b7be116d5560fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.347718 4672 tablet_bootstrap.cc:654] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.347757 4673 tablet_bootstrap.cc:654] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.349164 4673 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:09.349238 4673 ts_tablet_manager.cc:1367] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:09.349323 4676 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:09.349637 3717 tablet_service.cc:1467] Processing CreateTablet for tablet 49afc417528049839897ee5600b954e8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:09.349691 4673 raft_consensus.cc:348] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.349790 4673 raft_consensus.cc:374] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.349820 4673 raft_consensus.cc:729] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.349886 3717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49afc417528049839897ee5600b954e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:09.349965 4673 consensus_queue.cc:260] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.350121 4676 tablet_bootstrap.cc:654] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.350303 4673 ts_tablet_manager.cc:1398] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:09.350472 4673 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: Bootstrap starting.
I20240503 13:59:09.351476 4673 tablet_bootstrap.cc:654] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.351756 4676 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:09.351845 4676 ts_tablet_manager.cc:1367] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:09.352317 4676 raft_consensus.cc:348] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.352435 4676 raft_consensus.cc:374] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.352465 4676 raft_consensus.cc:729] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.352599 4676 consensus_queue.cc:260] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.352900 4676 ts_tablet_manager.cc:1398] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.353085 4676 tablet_bootstrap.cc:492] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:09.353284 4673 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: No bootstrap required, opened a new log
I20240503 13:59:09.353375 4673 ts_tablet_manager.cc:1367] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:09.353967 4676 tablet_bootstrap.cc:654] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.354171 4673 raft_consensus.cc:348] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.354279 4673 raft_consensus.cc:374] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.354314 4673 raft_consensus.cc:729] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.354456 4673 consensus_queue.cc:260] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.354749 4673 ts_tablet_manager.cc:1398] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.355604 4676 tablet_bootstrap.cc:492] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:09.355687 4676 ts_tablet_manager.cc:1367] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:09.356052 4672 tablet_bootstrap.cc:492] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc: No bootstrap required, opened a new log
I20240503 13:59:09.356148 4672 ts_tablet_manager.cc:1367] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:59:09.356132 4676 raft_consensus.cc:348] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.356235 4676 raft_consensus.cc:374] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.356274 4676 raft_consensus.cc:729] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.356431 4676 consensus_queue.cc:260] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.356576 4672 raft_consensus.cc:348] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.356699 4672 raft_consensus.cc:374] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.356734 4672 raft_consensus.cc:729] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.356766 4676 ts_tablet_manager.cc:1398] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.356868 4672 consensus_queue.cc:260] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.357139 4672 ts_tablet_manager.cc:1398] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:09.357782 4676 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Bootstrap starting.
I20240503 13:59:09.359275 4676 tablet_bootstrap.cc:654] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:09.360661 4676 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: No bootstrap required, opened a new log
I20240503 13:59:09.360741 4676 ts_tablet_manager.cc:1367] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:09.361130 4676 raft_consensus.cc:348] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.361227 4676 raft_consensus.cc:374] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:09.361258 4676 raft_consensus.cc:729] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Initialized, Role: FOLLOWER
I20240503 13:59:09.361389 4676 consensus_queue.cc:260] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.361620 4676 ts_tablet_manager.cc:1398] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:09.407675 4544 raft_consensus.cc:482] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.407799 4544 raft_consensus.cc:504] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.408124 4544 leader_election.cc:288] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:09.408550 3606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e7fa1300123438896b7be116d5560fa" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc" is_pre_election: true
I20240503 13:59:09.408687 3606 raft_consensus.cc:2454] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:09.408982 3673 leader_election.cc:302] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:09.409263 4544 raft_consensus.cc:2781] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.409337 4544 raft_consensus.cc:482] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.409374 4544 raft_consensus.cc:3037] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.410250 4544 raft_consensus.cc:504] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.410552 4544 leader_election.cc:288] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:09.410949 3606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e7fa1300123438896b7be116d5560fa" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:09.411062 3606 raft_consensus.cc:3037] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.411896 3606 raft_consensus.cc:2454] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:09.412200 3673 leader_election.cc:302] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:09.412437 4544 raft_consensus.cc:2781] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.412544 4544 raft_consensus.cc:686] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:09.412698 4544 consensus_queue.cc:237] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.414521 3458 catalog_manager.cc:5484] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:09.441407 4544 raft_consensus.cc:482] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.441576 4544 raft_consensus.cc:504] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.442011 4544 leader_election.cc:288] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:09.442549 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49afc417528049839897ee5600b954e8" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" is_pre_election: true
I20240503 13:59:09.442693 3869 raft_consensus.cc:2454] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 0.
I20240503 13:59:09.443008 3674 leader_election.cc:302] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:09.443220 4544 raft_consensus.cc:2781] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.443276 4544 raft_consensus.cc:482] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.443302 4544 raft_consensus.cc:3037] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.444149 4544 raft_consensus.cc:504] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.444461 4544 leader_election.cc:288] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Requested vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:09.444904 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49afc417528049839897ee5600b954e8" candidate_uuid: "3812cd29374a4b3485883810d0774867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
I20240503 13:59:09.445024 3869 raft_consensus.cc:3037] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.445852 3869 raft_consensus.cc:2454] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3812cd29374a4b3485883810d0774867 in term 1.
I20240503 13:59:09.446197 3674 leader_election.cc:302] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:09.446394 4544 raft_consensus.cc:2781] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.446480 4544 raft_consensus.cc:686] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Leader. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:09.446615 4544 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } }
I20240503 13:59:09.448371 3458 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3812cd29374a4b3485883810d0774867 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:09.508340 4714 raft_consensus.cc:482] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:09.508496 4714 raft_consensus.cc:504] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.508915 4714 leader_election.cc:288] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:09.509346 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fbc3876d064aa0a8e7f8f71fb3f35f" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867" is_pre_election: true
I20240503 13:59:09.509491 3739 raft_consensus.cc:2454] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 0.
I20240503 13:59:09.509811 3803 leader_election.cc:302] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:09.510043 4714 raft_consensus.cc:2781] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:09.510121 4714 raft_consensus.cc:482] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:09.510174 4714 raft_consensus.cc:3037] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.511230 4714 raft_consensus.cc:504] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.511562 4714 leader_election.cc:288] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:09.511994 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fbc3876d064aa0a8e7f8f71fb3f35f" candidate_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:09.512126 3739 raft_consensus.cc:3037] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:09.513046 3739 raft_consensus.cc:2454] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6ca23f9d9b4c91a6347c0924674bdd in term 1.
I20240503 13:59:09.513383 3803 leader_election.cc:302] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:09.513548 4714 raft_consensus.cc:2781] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:09.513639 4714 raft_consensus.cc:686] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Leader. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:09.513787 4714 consensus_queue.cc:237] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:09.515642 3458 catalog_manager.cc:5484] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131). New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:09.661623 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:09.661927 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:09.665216 4730 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 13:59:09.750443 3739 raft_consensus.cc:1261] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.751192 4565 consensus_queue.cc:1035] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.751978 3606 raft_consensus.cc:1261] T 2e7fa1300123438896b7be116d5560fa P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.752251 3869 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:09.752791 4500 consensus_queue.cc:1035] T 2e7fa1300123438896b7be116d5560fa P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:09.753099 4500 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 95.4139 ms
time per row: 0.047707 ms
I20240503 13:59:09.805051 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:36697
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:43017
--webserver_port=43571
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:09.918996 4762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:09.919272 4762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:09.919332 4762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:09.922919 4762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:09.922998 4762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:09.923082 4762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:09.926993 4762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:36697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=43571
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.4762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:09.928041 4762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:09.929281 4762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:09.938856 4767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:09.939577 4770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:09.940328 4768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:09.941361 4762 server_base.cc:1034] running on GCE node
I20240503 13:59:09.941854 4762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:09.942560 4762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:09.943745 4762 hybrid_clock.cc:648] HybridClock initialized: now 1714744749943732 us; error 30 us; skew 500 ppm
I20240503 13:59:09.946022 4762 webserver.cc:460] Webserver started at http://127.0.99.132:43571/ using document root <none> and password file <none>
I20240503 13:59:09.946637 4762 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:09.946735 4762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:09.950687 4762 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:09.952948 4776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:09.954187 4762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:09.954322 4762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4e553631c0564b078cefd1cd1cdcc90d"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:09.954949 4762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:09.975760 4762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:09.976509 4762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:09.976691 4762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:09.977294 4762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:09.978313 4762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:09.978368 4762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:09.978427 4762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:09.978453 4762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:09.987243 4762 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36697
I20240503 13:59:09.987269 4889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36697 every 8 connection(s)
I20240503 13:59:09.988408 4762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:09.991168 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 4762
I20240503 13:59:09.991626 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:39785
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:43017
--webserver_port=39521
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:09.998596 4890 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:09.998867 4890 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:09.999465 4890 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:10.000690 3464 ts_manager.cc:194] Re-registered known tserver with Master: 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697)
I20240503 13:59:10.001284 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:38015
W20240503 13:59:10.107614 4893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:10.107873 4893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:10.107937 4893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:10.111145 4893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:10.111228 4893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:10.111317 4893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:10.115290 4893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:39785
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=39521
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.4893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:10.116478 4893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:10.117764 4893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:10.127005 4898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:10.127518 4899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:10.128319 4901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:10.129010 4893 server_base.cc:1034] running on GCE node
I20240503 13:59:10.129455 4893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:10.130126 4893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:10.131297 4893 hybrid_clock.cc:648] HybridClock initialized: now 1714744750131266 us; error 47 us; skew 500 ppm
I20240503 13:59:10.133422 4893 webserver.cc:460] Webserver started at http://127.0.99.133:39521/ using document root <none> and password file <none>
I20240503 13:59:10.134032 4893 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:10.134115 4893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:10.137778 4893 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:10.139997 4907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.141230 4893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:10.141351 4893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:10.141830 4893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:10.156718 4893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:10.157454 4893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:10.157642 4893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:10.158401 4893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:10.159409 4893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:10.159461 4893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.159503 4893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:10.159524 4893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.168555 4893 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39785
I20240503 13:59:10.168586 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39785 every 8 connection(s)
I20240503 13:59:10.169692 4893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:10.178087 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 4893
I20240503 13:59:10.178580 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:43797
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:43017
--webserver_port=40553
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:10.179154 5021 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:10.179404 5021 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:10.180017 5021 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:10.181188 3464 ts_manager.cc:194] Re-registered known tserver with Master: 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785)
I20240503 13:59:10.181938 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:57947
W20240503 13:59:10.294886 5024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:10.295137 5024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:10.295183 5024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:10.298434 5024 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:10.298527 5024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:10.298621 5024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:10.302903 5024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:43797
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=40553
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.5024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:10.303956 5024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:10.305151 5024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:10.314882 5030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:10.315680 5032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:10.316185 5029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:10.317107 5024 server_base.cc:1034] running on GCE node
I20240503 13:59:10.317564 5024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:10.318298 5024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:10.319765 5024 hybrid_clock.cc:648] HybridClock initialized: now 1714744750319747 us; error 32 us; skew 500 ppm
I20240503 13:59:10.322077 5024 webserver.cc:460] Webserver started at http://127.0.99.134:40553/ using document root <none> and password file <none>
I20240503 13:59:10.322736 5024 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:10.322819 5024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:10.326651 5024 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 13:59:10.329046 5038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.330313 5024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:10.330446 5024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:10.330936 5024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:10.344447 5024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:10.345218 5024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:10.345389 5024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:10.346166 5024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:10.347470 5045 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 13:59:10.350944 5024 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 13:59:10.351025 5024 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240503 13:59:10.351105 5024 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 13:59:10.352802 5024 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 13:59:10.352855 5024 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 13:59:10.353122 5045 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap starting.
I20240503 13:59:10.363317 5024 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43797
I20240503 13:59:10.363332 5152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43797 every 8 connection(s)
I20240503 13:59:10.364527 5024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:10.365013 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 5024
I20240503 13:59:10.365468 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:43643
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:43017
--webserver_port=39383
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:10.366766 5045 log.cc:824] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:10.368994 5045 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:10.369364 5045 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap complete.
I20240503 13:59:10.370541 5045 ts_tablet_manager.cc:1367] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20240503 13:59:10.373647 5045 raft_consensus.cc:348] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:10.374043 5045 raft_consensus.cc:729] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Initialized, Role: FOLLOWER
I20240503 13:59:10.374667 5045 consensus_queue.cc:260] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } }
I20240503 13:59:10.375566 5045 ts_tablet_manager.cc:1398] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240503 13:59:10.375758 5153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:10.376010 5153 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:10.376749 5153 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:10.378882 3464 ts_manager.cc:194] Re-registered known tserver with Master: 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797)
I20240503 13:59:10.379686 3464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:48015
I20240503 13:59:10.382069 5153 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
W20240503 13:59:10.482899 5155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:10.483150 5155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:10.483192 5155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:10.486475 5155 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:10.486563 5155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:10.486635 5155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:10.490633 5155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:43643
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=39383
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.5155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:10.491662 5155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:10.492888 5155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:10.502179 5163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:10.502828 5166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:10.503448 5164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:10.504356 5155 server_base.cc:1034] running on GCE node
I20240503 13:59:10.504774 5155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:10.505379 5155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:10.506525 5155 hybrid_clock.cc:648] HybridClock initialized: now 1714744750506508 us; error 33 us; skew 500 ppm
I20240503 13:59:10.508535 5155 webserver.cc:460] Webserver started at http://127.0.99.135:39383/ using document root <none> and password file <none>
I20240503 13:59:10.509096 5155 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:10.509164 5155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:10.512754 5155 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:10.514935 5172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.516110 5155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:10.516218 5155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "07bdad76c7bd40afb3147833b3305924"
format_stamp: "Formatted at 2024-05-03 13:59:02 on dist-test-slave-6fwv"
I20240503 13:59:10.516690 5155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:10.538161 5155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:10.538983 5155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:10.539171 5155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:10.539825 5155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:10.540846 5155 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:10.540906 5155 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.540961 5155 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:10.540988 5155 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:10.551424 5155 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43643
I20240503 13:59:10.551450 5291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43643 every 8 connection(s)
I20240503 13:59:10.552465 5155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:10.560899 5292 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:10.561133 5292 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:10.561719 5292 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:10.562418 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 5155
I20240503 13:59:10.562628 398 test_util.cc:268] Using random seed: 532471106
I20240503 13:59:10.562786 3458 ts_manager.cc:194] Re-registered known tserver with Master: 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643)
I20240503 13:59:10.563383 3458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:45953
I20240503 13:59:10.571578 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 13:59:10.571780 398 test_util.cc:268] Using random seed: 532480259
I20240503 13:59:10.580397 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 13:59:10.580559 398 test_util.cc:268] Using random seed: 532489035
I20240503 13:59:10.594332 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 13:59:10.594511 398 test_util.cc:268] Using random seed: 532502990
I20240503 13:59:10.613081 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 13:59:10.613257 398 test_util.cc:268] Using random seed: 532521734
I20240503 13:59:10.644109 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 13:59:11.003615 4890 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
I20240503 13:59:11.119843 5217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.130290 5087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.146277 3849 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:11.149559 3717 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:11.156060 4823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.172399 3587 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:11.176865 4955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.184355 5021 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
I20240503 13:59:11.540380 4955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.545166 5217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.557869 3849 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:11.563506 4823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.565659 5292 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
I20240503 13:59:11.581853 3717 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:11.601550 5087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:11.622519 3587 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:11.717851 3738 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:11.722599 3869 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240503 13:59:11.723004 5379 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 13:59:11.726850 5367 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:11.729095 3870 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
W20240503 13:59:11.732161 3675 consensus_peers.cc:485] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 -> Peer 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785): Couldn't send request to peer 14a79ecc8c2c45adac20a5239cd45fd2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19ebb671e56f4e29adb6a2c0a7c6b00e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:11.733753 3458 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index -1 to 37, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.749768 3870 consensus_queue.cc:237] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:11.751864 3603 raft_consensus.cc:1261] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 13:59:11.752233 4622 consensus_queue.cc:1035] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 13:59:11.756681 4622 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index -1 to 32, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:11.757197 3604 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index -1 to 32, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:11.759596 3464 catalog_manager.cc:5484] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 32, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:11.764531 3803 consensus_peers.cc:485] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697): Couldn't send request to peer 4e553631c0564b078cefd1cd1cdcc90d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37a9c633962b40f1b42376cf597ad415. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:11.781390 3870 consensus_queue.cc:237] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:11.783025 3737 raft_consensus.cc:1261] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240503 13:59:11.784294 5377 consensus_queue.cc:1035] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 43, Time since last communication: 0.000s
I20240503 13:59:11.785916 4622 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:11.787009 3737 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:11.789316 3464 catalog_manager.cc:5484] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 45, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:11.793390 3803 consensus_peers.cc:485] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643): Couldn't send request to peer 07bdad76c7bd40afb3147833b3305924. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc8c685a1dd4496b9be2b1e09d237371. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:11.809525 3737 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:11.814757 3869 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 13:59:11.815222 5490 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 13:59:11.818210 5379 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
W20240503 13:59:11.819693 3674 consensus_peers.cc:485] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49afc417528049839897ee5600b954e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:11.821171 3458 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index -1 to 25, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.823817 3870 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:11.838498 3870 consensus_queue.cc:237] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
W20240503 13:59:11.845714 3806 consensus_peers.cc:485] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785): Couldn't send request to peer 14a79ecc8c2c45adac20a5239cd45fd2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62fbc3876d064aa0a8e7f8f71fb3f35f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:11.851397 3737 raft_consensus.cc:1261] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 13:59:11.853055 5377 consensus_queue.cc:1035] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 13:59:11.874711 5348 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:11.880898 3738 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:11.880910 3464 catalog_manager.cc:5484] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 25, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.918972 3738 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:11.920338 3606 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 13:59:11.920686 5490 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20240503 13:59:11.922708 5380 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:11.925766 3464 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index -1 to 46, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.926913 3603 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
W20240503 13:59:11.930383 3672 consensus_peers.cc:485] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 -> Peer 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697): Couldn't send request to peer 4e553631c0564b078cefd1cd1cdcc90d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd23523f46044bb7b7e4f44056687c43. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:11.953066 3870 consensus_queue.cc:237] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:11.956912 5107 raft_consensus.cc:1261] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:11.957417 5450 consensus_queue.cc:1035] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:11.959659 4622 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:11.962931 5107 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:11.965073 3464 catalog_manager.cc:5484] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 2, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:11.969372 3803 consensus_peers.cc:485] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643): Couldn't send request to peer 07bdad76c7bd40afb3147833b3305924. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8df17b9134914ca3a2c04800435b3697. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.010669 3737 consensus_queue.cc:237] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
W20240503 13:59:12.013639 3674 consensus_peers.cc:485] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1c58189e2df41c3b8b4a0924ae42a03. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.021268 3604 raft_consensus.cc:1261] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 13:59:12.025676 5490 consensus_queue.cc:1035] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 13:59:12.030685 5380 raft_consensus.cc:2932] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:12.033492 3464 catalog_manager.cc:5484] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index -1 to 55, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.037397 3604 raft_consensus.cc:2932] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:12.079800 3869 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
W20240503 13:59:12.082726 3803 consensus_peers.cc:485] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697): Couldn't send request to peer 4e553631c0564b078cefd1cd1cdcc90d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e572db34c8844384b728068168645ecc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.083180 3738 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 13:59:12.087098 5450 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240503 13:59:12.098990 5348 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:12.099649 3738 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:12.102149 3464 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 55, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.156275 3870 consensus_queue.cc:237] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
W20240503 13:59:12.158399 3806 consensus_peers.cc:485] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785): Couldn't send request to peer 14a79ecc8c2c45adac20a5239cd45fd2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 809441335aa64f959ef084cb3c8b2434. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.160698 3603 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240503 13:59:12.161163 5377 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 13:59:12.163946 5348 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:12.164408 3603 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:12.166635 3464 catalog_manager.cc:5484] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 63, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.181599 5537 ts_tablet_manager.cc:902] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.182729 5537 tablet_copy_client.cc:308] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.194497 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet 37a9c633962b40f1b42376cf597ad415 from peer 4e553631c0564b078cefd1cd1cdcc90d ({username='slave'} at 127.0.99.132:52411)
I20240503 13:59:12.194674 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet 37a9c633962b40f1b42376cf597ad415 from peer 4e553631c0564b078cefd1cd1cdcc90d at {username='slave'} at 127.0.99.132:52411: session id = 4e553631c0564b078cefd1cd1cdcc90d-37a9c633962b40f1b42376cf597ad415
I20240503 13:59:12.196168 3890 tablet_copy_source_session.cc:205] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.205950 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a9c633962b40f1b42376cf597ad415. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.215041 5543 ts_tablet_manager.cc:902] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:12.224576 5537 tablet_copy_client.cc:783] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.224823 5537 tablet_copy_client.cc:647] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.226984 5543 tablet_copy_client.cc:308] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:12.231719 3604 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:12.234412 5537 tablet_copy_client.cc:523] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.236375 3738 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 13:59:12.238757 5537 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap starting.
I20240503 13:59:12.238737 5318 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 67, Time since last communication: 0.000s
I20240503 13:59:12.240391 4490 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:12.241079 3738 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:12.244002 3458 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index -1 to 69, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.258987 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e from peer 14a79ecc8c2c45adac20a5239cd45fd2 ({username='slave'} at 127.0.99.133:34067)
I20240503 13:59:12.259173 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet 19ebb671e56f4e29adb6a2c0a7c6b00e from peer 14a79ecc8c2c45adac20a5239cd45fd2 at {username='slave'} at 127.0.99.133:34067: session id = 14a79ecc8c2c45adac20a5239cd45fd2-19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:12.260664 3759 tablet_copy_source_session.cc:205] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 13:59:12.266345 3541 consensus_peers.cc:485] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc -> Peer 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643): Couldn't send request to peer 07bdad76c7bd40afb3147833b3305924. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97d59e7bd0da44d9a7af791f2477b183. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.267689 5537 log.cc:824] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:12.271481 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.280467 5543 tablet_copy_client.cc:783] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.280858 5543 tablet_copy_client.cc:647] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.293141 5556 ts_tablet_manager.cc:902] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.307097 3869 consensus_queue.cc:237] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
W20240503 13:59:12.308971 3805 consensus_peers.cc:485] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcf42adb4ddc4a25907a5c5fa94e31b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.310343 3607 raft_consensus.cc:1261] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 13:59:12.311211 5556 tablet_copy_client.cc:308] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.315501 5348 consensus_queue.cc:1035] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 13:59:12.316331 5543 tablet_copy_client.cc:523] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.318933 5543 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap starting.
I20240503 13:59:12.324290 4622 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:12.328496 3464 catalog_manager.cc:5484] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index -1 to 62, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.331164 3603 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:12.331785 5543 log.cc:824] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:12.333737 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet fc8c685a1dd4496b9be2b1e09d237371 from peer 07bdad76c7bd40afb3147833b3305924 ({username='slave'} at 127.0.99.135:48391)
I20240503 13:59:12.333855 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet fc8c685a1dd4496b9be2b1e09d237371 from peer 07bdad76c7bd40afb3147833b3305924 at {username='slave'} at 127.0.99.135:48391: session id = 07bdad76c7bd40afb3147833b3305924-fc8c685a1dd4496b9be2b1e09d237371
I20240503 13:59:12.334833 3890 tablet_copy_source_session.cc:205] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.416137 5543 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=62 overwritten=0 applied=62 ignored=0} inserts{seen=693 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.416554 5543 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap complete.
I20240503 13:59:12.416957 5543 ts_tablet_manager.cc:1367] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent bootstrapping tablet: real 0.098s user 0.063s sys 0.011s
I20240503 13:59:12.419140 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc8c685a1dd4496b9be2b1e09d237371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.419821 5543 raft_consensus.cc:348] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:12.420087 5543 raft_consensus.cc:729] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14a79ecc8c2c45adac20a5239cd45fd2, State: Initialized, Role: LEARNER
I20240503 13:59:12.420521 5543 consensus_queue.cc:260] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 62, Last appended: 1.62, Last appended by leader: 62, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:12.421666 5543 ts_tablet_manager.cc:1398] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240503 13:59:12.421675 3607 consensus_queue.cc:237] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.429041 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-19ebb671e56f4e29adb6a2c0a7c6b00e received from {username='slave'} at 127.0.99.133:34067
I20240503 13:59:12.429164 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-19ebb671e56f4e29adb6a2c0a7c6b00e on tablet 19ebb671e56f4e29adb6a2c0a7c6b00e with peer 14a79ecc8c2c45adac20a5239cd45fd2
W20240503 13:59:12.429636 3541 consensus_peers.cc:485] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc -> Peer 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697): Couldn't send request to peer 4e553631c0564b078cefd1cd1cdcc90d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adfa836cb94c4050bdb65e007b232f2b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.430498 5568 ts_tablet_manager.cc:902] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:12.430951 5543 ts_tablet_manager.cc:902] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.432134 5543 tablet_copy_client.cc:308] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.432768 5568 tablet_copy_client.cc:308] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:12.434419 3869 raft_consensus.cc:1261] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:12.436281 5384 consensus_queue.cc:1035] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 13:59:12.436381 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f from peer 14a79ecc8c2c45adac20a5239cd45fd2 ({username='slave'} at 127.0.99.133:48411)
I20240503 13:59:12.436470 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet 62fbc3876d064aa0a8e7f8f71fb3f35f from peer 14a79ecc8c2c45adac20a5239cd45fd2 at {username='slave'} at 127.0.99.133:48411: session id = 14a79ecc8c2c45adac20a5239cd45fd2-62fbc3876d064aa0a8e7f8f71fb3f35f
I20240503 13:59:12.437425 3890 tablet_copy_source_session.cc:205] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.438499 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.441738 5318 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:12.444535 3458 catalog_manager.cc:5484] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index -1 to 74, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.446010 5543 tablet_copy_client.cc:783] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.446183 5543 tablet_copy_client.cc:647] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.448171 3870 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } } }
I20240503 13:59:12.455626 5537 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=54 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:12.456142 5537 tablet_bootstrap.cc:492] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap complete.
I20240503 13:59:12.456501 5537 ts_tablet_manager.cc:1367] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Time spent bootstrapping tablet: real 0.219s user 0.075s sys 0.001s
I20240503 13:59:12.459471 5537 raft_consensus.cc:348] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.461062 5537 raft_consensus.cc:729] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e553631c0564b078cefd1cd1cdcc90d, State: Initialized, Role: LEARNER
I20240503 13:59:12.461486 5537 consensus_queue.cc:260] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.462481 5537 ts_tablet_manager.cc:1398] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:59:12.464200 5543 tablet_copy_client.cc:523] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.464758 5585 ts_tablet_manager.cc:902] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:12.465608 5585 tablet_copy_client.cc:308] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:12.465852 5543 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap starting.
I20240503 13:59:12.469861 5556 tablet_copy_client.cc:783] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.470116 5556 tablet_copy_client.cc:647] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.473385 5580 ts_tablet_manager.cc:902] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:12.474011 5580 tablet_copy_client.cc:308] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:12.479090 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet bd23523f46044bb7b7e4f44056687c43 from peer 4e553631c0564b078cefd1cd1cdcc90d ({username='slave'} at 127.0.99.132:55507)
I20240503 13:59:12.479233 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet bd23523f46044bb7b7e4f44056687c43 from peer 4e553631c0564b078cefd1cd1cdcc90d at {username='slave'} at 127.0.99.132:55507: session id = 4e553631c0564b078cefd1cd1cdcc90d-bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:12.480154 3759 tablet_copy_source_session.cc:205] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.481053 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd23523f46044bb7b7e4f44056687c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.483539 5585 tablet_copy_client.cc:783] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.483716 5585 tablet_copy_client.cc:647] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.483827 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-37a9c633962b40f1b42376cf597ad415 received from {username='slave'} at 127.0.99.132:52411
I20240503 13:59:12.483942 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-37a9c633962b40f1b42376cf597ad415 on tablet 37a9c633962b40f1b42376cf597ad415 with peer 4e553631c0564b078cefd1cd1cdcc90d
I20240503 13:59:12.490613 3758 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet c1c58189e2df41c3b8b4a0924ae42a03 from peer 6d1a97f876fc4f8698f737fcf1b782ad ({username='slave'} at 127.0.99.134:36473)
I20240503 13:59:12.490763 3758 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet c1c58189e2df41c3b8b4a0924ae42a03 from peer 6d1a97f876fc4f8698f737fcf1b782ad at {username='slave'} at 127.0.99.134:36473: session id = 6d1a97f876fc4f8698f737fcf1b782ad-c1c58189e2df41c3b8b4a0924ae42a03
I20240503 13:59:12.491724 3758 tablet_copy_source_session.cc:205] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.492151 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet 49afc417528049839897ee5600b954e8 from peer 6d1a97f876fc4f8698f737fcf1b782ad ({username='slave'} at 127.0.99.134:36473)
I20240503 13:59:12.492231 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet 49afc417528049839897ee5600b954e8 from peer 6d1a97f876fc4f8698f737fcf1b782ad at {username='slave'} at 127.0.99.134:36473: session id = 6d1a97f876fc4f8698f737fcf1b782ad-49afc417528049839897ee5600b954e8
I20240503 13:59:12.492051 5556 tablet_copy_client.cc:523] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.493038 3759 tablet_copy_source_session.cc:205] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.495838 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49afc417528049839897ee5600b954e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.500792 5568 tablet_copy_client.cc:783] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.501091 5568 tablet_copy_client.cc:647] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.502135 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c58189e2df41c3b8b4a0924ae42a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.504050 5585 tablet_copy_client.cc:523] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.505610 5585 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap starting.
I20240503 13:59:12.506896 5580 tablet_copy_client.cc:783] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.507118 5580 tablet_copy_client.cc:647] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.511281 5568 tablet_copy_client.cc:523] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.511474 5537 ts_tablet_manager.cc:902] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.511924 5537 tablet_copy_client.cc:308] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.513679 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet e572db34c8844384b728068168645ecc from peer 4e553631c0564b078cefd1cd1cdcc90d ({username='slave'} at 127.0.99.132:52411)
I20240503 13:59:12.513811 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet e572db34c8844384b728068168645ecc from peer 4e553631c0564b078cefd1cd1cdcc90d at {username='slave'} at 127.0.99.132:52411: session id = 4e553631c0564b078cefd1cd1cdcc90d-e572db34c8844384b728068168645ecc
I20240503 13:59:12.514012 5568 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap starting.
I20240503 13:59:12.514780 3890 tablet_copy_source_session.cc:205] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.515658 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572db34c8844384b728068168645ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.519665 5580 tablet_copy_client.cc:523] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.520746 5537 tablet_copy_client.cc:783] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.520926 5537 tablet_copy_client.cc:647] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.523608 5556 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Bootstrap starting.
I20240503 13:59:12.526297 5580 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap starting.
I20240503 13:59:12.529947 5600 ts_tablet_manager.cc:902] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.530640 5600 tablet_copy_client.cc:308] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.532130 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet 8df17b9134914ca3a2c04800435b3697 from peer 07bdad76c7bd40afb3147833b3305924 ({username='slave'} at 127.0.99.135:48391)
I20240503 13:59:12.532253 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet 8df17b9134914ca3a2c04800435b3697 from peer 07bdad76c7bd40afb3147833b3305924 at {username='slave'} at 127.0.99.135:48391: session id = 07bdad76c7bd40afb3147833b3305924-8df17b9134914ca3a2c04800435b3697
I20240503 13:59:12.533198 3890 tablet_copy_source_session.cc:205] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.537299 5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df17b9134914ca3a2c04800435b3697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.548126 5600 tablet_copy_client.cc:783] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.548331 5600 tablet_copy_client.cc:647] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.553503 5537 tablet_copy_client.cc:523] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.555058 5537 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap starting.
I20240503 13:59:12.566570 5600 tablet_copy_client.cc:523] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.568272 5600 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: Bootstrap starting.
I20240503 13:59:12.585865 5543 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=725 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.586230 5543 tablet_bootstrap.cc:492] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap complete.
I20240503 13:59:12.586458 5543 ts_tablet_manager.cc:1367] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent bootstrapping tablet: real 0.121s user 0.053s sys 0.001s
I20240503 13:59:12.587127 5543 raft_consensus.cc:348] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:12.587539 5543 raft_consensus.cc:729] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14a79ecc8c2c45adac20a5239cd45fd2, State: Initialized, Role: LEARNER
I20240503 13:59:12.587539 3738 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:12.587659 5543 consensus_queue.cc:260] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
W20240503 13:59:12.589022 3674 consensus_peers.cc:485] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 -> Peer 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): Couldn't send request to peer 6d1a97f876fc4f8698f737fcf1b782ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e967e4257bda4e4fa9b3fdfadd22368f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:12.578943 5556 log.cc:824] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:12.589362 3604 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 13:59:12.591917 4756 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 13:59:12.597497 5380 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:12.600616 3464 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index -1 to 81, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.600694 3604 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } } }
I20240503 13:59:12.610229 5543 ts_tablet_manager.cc:1398] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:59:12.613373 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-62fbc3876d064aa0a8e7f8f71fb3f35f received from {username='slave'} at 127.0.99.133:48411
I20240503 13:59:12.613479 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-62fbc3876d064aa0a8e7f8f71fb3f35f on tablet 62fbc3876d064aa0a8e7f8f71fb3f35f with peer 14a79ecc8c2c45adac20a5239cd45fd2
I20240503 13:59:12.618827 5585 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap replayed 1/1 log segments. Stats: ops{read=62 overwritten=0 applied=62 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.619258 5585 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap complete.
I20240503 13:59:12.619537 5585 ts_tablet_manager.cc:1367] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: Time spent bootstrapping tablet: real 0.114s user 0.056s sys 0.013s
I20240503 13:59:12.620321 5585 raft_consensus.cc:348] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.620543 5585 raft_consensus.cc:729] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e553631c0564b078cefd1cd1cdcc90d, State: Initialized, Role: LEARNER
I20240503 13:59:12.620713 5585 consensus_queue.cc:260] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 62, Last appended: 1.62, Last appended by leader: 62, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.621152 5585 ts_tablet_manager.cc:1398] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:12.624688 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-bd23523f46044bb7b7e4f44056687c43 received from {username='slave'} at 127.0.99.132:55507
I20240503 13:59:12.624794 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-bd23523f46044bb7b7e4f44056687c43 on tablet bd23523f46044bb7b7e4f44056687c43 with peer 4e553631c0564b078cefd1cd1cdcc90d
I20240503 13:59:12.629093 5600 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.629552 5600 tablet_bootstrap.cc:492] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: Bootstrap complete.
I20240503 13:59:12.630020 5600 ts_tablet_manager.cc:1367] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.007s
I20240503 13:59:12.633044 5600 raft_consensus.cc:348] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:12.633464 5600 raft_consensus.cc:729] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07bdad76c7bd40afb3147833b3305924, State: Initialized, Role: LEARNER
I20240503 13:59:12.634033 5600 consensus_queue.cc:260] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:12.635197 5600 ts_tablet_manager.cc:1398] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:12.639230 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 07bdad76c7bd40afb3147833b3305924-8df17b9134914ca3a2c04800435b3697 received from {username='slave'} at 127.0.99.135:48391
I20240503 13:59:12.639339 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 07bdad76c7bd40afb3147833b3305924-8df17b9134914ca3a2c04800435b3697 on tablet 8df17b9134914ca3a2c04800435b3697 with peer 07bdad76c7bd40afb3147833b3305924
I20240503 13:59:12.672709 4844 raft_consensus.cc:1203] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.31->[1.32-1.69] Dedup: 1.55->[1.56-1.69]
I20240503 13:59:12.683019 5556 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.683375 5556 tablet_bootstrap.cc:492] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Bootstrap complete.
I20240503 13:59:12.683593 5556 ts_tablet_manager.cc:1367] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Time spent bootstrapping tablet: real 0.162s user 0.067s sys 0.008s
I20240503 13:59:12.684213 5556 raft_consensus.cc:348] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:12.684365 5556 raft_consensus.cc:729] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07bdad76c7bd40afb3147833b3305924, State: Initialized, Role: LEARNER
I20240503 13:59:12.684470 5556 consensus_queue.cc:260] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:12.686460 5556 ts_tablet_manager.cc:1398] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:12.689770 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 07bdad76c7bd40afb3147833b3305924-fc8c685a1dd4496b9be2b1e09d237371 received from {username='slave'} at 127.0.99.135:48391
I20240503 13:59:12.689878 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 07bdad76c7bd40afb3147833b3305924-fc8c685a1dd4496b9be2b1e09d237371 on tablet fc8c685a1dd4496b9be2b1e09d237371 with peer 07bdad76c7bd40afb3147833b3305924
I20240503 13:59:12.711771 5556 ts_tablet_manager.cc:902] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: Initiating tablet copy from peer bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:12.712289 5556 tablet_copy_client.cc:308] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43281
I20240503 13:59:12.719111 3628 tablet_copy_service.cc:140] P bc41cca530f54259aca15e34563208dc: Received BeginTabletCopySession request for tablet 97d59e7bd0da44d9a7af791f2477b183 from peer 07bdad76c7bd40afb3147833b3305924 ({username='slave'} at 127.0.99.135:48765)
I20240503 13:59:12.719295 3628 tablet_copy_service.cc:161] P bc41cca530f54259aca15e34563208dc: Beginning new tablet copy session on tablet 97d59e7bd0da44d9a7af791f2477b183 from peer 07bdad76c7bd40afb3147833b3305924 at {username='slave'} at 127.0.99.135:48765: session id = 07bdad76c7bd40afb3147833b3305924-97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:12.734709 5348 raft_consensus.cc:1050] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d to VOTER
I20240503 13:59:12.735505 5348 consensus_queue.cc:237] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:12.740520 4843 raft_consensus.cc:1261] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:12.743132 5241 consensus_queue.cc:1035] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 13:59:12.743642 3628 tablet_copy_source_session.cc:205] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.744292 3604 raft_consensus.cc:1261] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:12.745889 5450 consensus_queue.cc:1035] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 13:59:12.746325 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d59e7bd0da44d9a7af791f2477b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.748373 5348 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 32 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.750360 5556 tablet_copy_client.cc:783] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.750532 5556 tablet_copy_client.cc:647] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.750813 3604 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 32 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.753703 3458 catalog_manager.cc:5484] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 32 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.756433 5242 raft_consensus.cc:1203] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.44->[1.45-1.68] Dedup: 1.61->[1.62-1.68]
I20240503 13:59:12.763865 5556 tablet_copy_client.cc:523] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.766232 4975 raft_consensus.cc:1203] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.73] Dedup: 1.62->[1.63-1.73]
I20240503 13:59:12.767709 5543 ts_tablet_manager.cc:902] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.768143 5543 tablet_copy_client.cc:308] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.769891 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet 809441335aa64f959ef084cb3c8b2434 from peer 14a79ecc8c2c45adac20a5239cd45fd2 ({username='slave'} at 127.0.99.133:48411)
I20240503 13:59:12.770028 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet 809441335aa64f959ef084cb3c8b2434 from peer 14a79ecc8c2c45adac20a5239cd45fd2 at {username='slave'} at 127.0.99.133:48411: session id = 14a79ecc8c2c45adac20a5239cd45fd2-809441335aa64f959ef084cb3c8b2434
I20240503 13:59:12.770968 3890 tablet_copy_source_session.cc:205] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.771973 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809441335aa64f959ef084cb3c8b2434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.780035 5556 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: Bootstrap starting.
I20240503 13:59:12.782119 5543 tablet_copy_client.cc:783] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.782359 5543 tablet_copy_client.cc:647] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.788647 5537 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=697 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.788952 5537 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap complete.
I20240503 13:59:12.789166 5537 ts_tablet_manager.cc:1367] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: Time spent bootstrapping tablet: real 0.234s user 0.058s sys 0.008s
I20240503 13:59:12.789683 5537 raft_consensus.cc:348] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.789824 5537 raft_consensus.cc:729] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e553631c0564b078cefd1cd1cdcc90d, State: Initialized, Role: LEARNER
I20240503 13:59:12.789938 5537 consensus_queue.cc:260] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:12.795377 3869 consensus_queue.cc:237] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:12.795105 4843 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 32 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.798902 4843 raft_consensus.cc:1261] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240503 13:59:12.799340 5348 consensus_queue.cc:1035] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 13:59:12.803647 5537 ts_tablet_manager.cc:1398] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240503 13:59:12.810317 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-e572db34c8844384b728068168645ecc received from {username='slave'} at 127.0.99.132:52411
I20240503 13:59:12.810449 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-e572db34c8844384b728068168645ecc on tablet e572db34c8844384b728068168645ecc with peer 4e553631c0564b078cefd1cd1cdcc90d
I20240503 13:59:12.811205 5450 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.813522 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37a9c633962b40f1b42376cf597ad415 with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:12.813848 4843 raft_consensus.cc:2932] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.817687 3458 catalog_manager.cc:5484] T 37a9c633962b40f1b42376cf597ad415 P 4e553631c0564b078cefd1cd1cdcc90d reported cstate change: config changed from index 74 to 75, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.822499 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a9c633962b40f1b42376cf597ad415 with delete_type TABLET_DATA_TOMBSTONED (TS bc41cca530f54259aca15e34563208dc not found in new config with opid_index 75) from {username='slave'} at 127.0.0.1:56666
I20240503 13:59:12.833360 5658 tablet_replica.cc:330] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: stopping tablet replica
I20240503 13:59:12.836728 4974 raft_consensus.cc:1203] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.50] Dedup: 1.40->[1.41-1.50]
I20240503 13:59:12.836992 5537 ts_tablet_manager.cc:902] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: Initiating tablet copy from peer bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:12.837505 5537 tablet_copy_client.cc:308] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43281
I20240503 13:59:12.838299 5658 raft_consensus.cc:2229] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:12.843853 3628 tablet_copy_service.cc:140] P bc41cca530f54259aca15e34563208dc: Received BeginTabletCopySession request for tablet adfa836cb94c4050bdb65e007b232f2b from peer 4e553631c0564b078cefd1cd1cdcc90d ({username='slave'} at 127.0.99.132:60531)
I20240503 13:59:12.844955 3628 tablet_copy_service.cc:161] P bc41cca530f54259aca15e34563208dc: Beginning new tablet copy session on tablet adfa836cb94c4050bdb65e007b232f2b from peer 4e553631c0564b078cefd1cd1cdcc90d at {username='slave'} at 127.0.99.132:60531: session id = 4e553631c0564b078cefd1cd1cdcc90d-adfa836cb94c4050bdb65e007b232f2b
I20240503 13:59:12.845257 4843 raft_consensus.cc:1203] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.71] Dedup: 1.62->[1.63-1.71]
I20240503 13:59:12.845772 5658 raft_consensus.cc:2258] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:12.846107 3628 tablet_copy_source_session.cc:205] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.854902 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfa836cb94c4050bdb65e007b232f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.856750 5568 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=49 ignored=0} inserts{seen=680 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:12.857203 5568 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap complete.
I20240503 13:59:12.857411 5537 tablet_copy_client.cc:783] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.857517 4756 raft_consensus.cc:1050] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 to VOTER
I20240503 13:59:12.857585 5537 tablet_copy_client.cc:647] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.858009 4756 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:12.858399 5568 ts_tablet_manager.cc:1367] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent bootstrapping tablet: real 0.344s user 0.062s sys 0.000s
I20240503 13:59:12.859061 5568 raft_consensus.cc:348] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:12.860240 4973 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 13:59:12.860687 5643 ts_tablet_manager.cc:902] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:12.861173 5379 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 13:59:12.861236 5643 tablet_copy_client.cc:308] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:12.863543 5658 ts_tablet_manager.cc:1875] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:12.865505 5568 raft_consensus.cc:729] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Initialized, Role: LEARNER
I20240503 13:59:12.865764 5568 consensus_queue.cc:260] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 1.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:12.866214 5658 ts_tablet_manager.cc:1888] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240503 13:59:12.866317 5658 log.cc:1196] T 37a9c633962b40f1b42376cf597ad415 P bc41cca530f54259aca15e34563208dc: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/37a9c633962b40f1b42376cf597ad415
I20240503 13:59:12.866467 5568 ts_tablet_manager.cc:1398] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 13:59:12.866982 3451 catalog_manager.cc:4830] TS bc41cca530f54259aca15e34563208dc (127.0.99.129:43281): tablet 37a9c633962b40f1b42376cf597ad415 (table rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]) successfully deleted
I20240503 13:59:12.867839 4975 raft_consensus.cc:1203] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.76] Dedup: 1.76->[]
I20240503 13:59:12.868208 5367 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 37 to 77, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.870600 4975 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 37 to 77, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.873641 3464 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 reported cstate change: config changed from index 37 to 77, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.875106 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet dcf42adb4ddc4a25907a5c5fa94e31b7 from peer 6d1a97f876fc4f8698f737fcf1b782ad ({username='slave'} at 127.0.99.134:40103)
I20240503 13:59:12.875231 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet dcf42adb4ddc4a25907a5c5fa94e31b7 from peer 6d1a97f876fc4f8698f737fcf1b782ad at {username='slave'} at 127.0.99.134:40103: session id = 6d1a97f876fc4f8698f737fcf1b782ad-dcf42adb4ddc4a25907a5c5fa94e31b7
I20240503 13:59:12.876174 3890 tablet_copy_source_session.cc:205] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:12.877156 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf42adb4ddc4a25907a5c5fa94e31b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:12.879055 5543 tablet_copy_client.cc:523] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.879058 3738 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:12.879937 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-49afc417528049839897ee5600b954e8 received from {username='slave'} at 127.0.99.134:36473
I20240503 13:59:12.880029 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-49afc417528049839897ee5600b954e8 on tablet 49afc417528049839897ee5600b954e8 with peer 6d1a97f876fc4f8698f737fcf1b782ad
I20240503 13:59:12.881997 5348 raft_consensus.cc:1050] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 07bdad76c7bd40afb3147833b3305924 to VOTER
I20240503 13:59:12.882421 5543 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap starting.
I20240503 13:59:12.882534 5348 consensus_queue.cc:237] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:12.885620 3869 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 13:59:12.887364 4973 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 13:59:12.888445 5379 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 13:59:12.890702 5490 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 77 to 79, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.891114 4974 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 77 to 79, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.892215 5580 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=742 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.892606 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e with cas_config_opid_index 77: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:12.893779 5377 raft_consensus.cc:1050] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 to VOTER
I20240503 13:59:12.894141 3458 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 reported cstate change: config changed from index 77 to 79, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.894224 5377 consensus_queue.cc:237] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:12.896044 4973 raft_consensus.cc:1261] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 13:59:12.896436 5348 consensus_queue.cc:1035] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240503 13:59:12.897644 5580 tablet_bootstrap.cc:492] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap complete.
I20240503 13:59:12.897929 5580 ts_tablet_manager.cc:1367] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent bootstrapping tablet: real 0.372s user 0.067s sys 0.010s
I20240503 13:59:12.898101 5242 raft_consensus.cc:1261] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 13:59:12.898593 5580 raft_consensus.cc:348] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:12.898864 5580 raft_consensus.cc:729] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Initialized, Role: LEARNER
I20240503 13:59:12.899024 5580 consensus_queue.cc:260] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:12.899418 5241 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index 25 to 51, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.900144 5580 ts_tablet_manager.cc:1398] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:12.900781 4973 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index 25 to 51, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.903213 3458 catalog_manager.cc:5484] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 25 to 51, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:12.904857 3737 raft_consensus.cc:1261] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240503 13:59:12.906234 3849 tablet_service.cc:1514] Processing DeleteTablet for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e with delete_type TABLET_DATA_TOMBSTONED (TS 0c6ca23f9d9b4c91a6347c0924674bdd not found in new config with opid_index 79) from {username='slave'} at 127.0.0.1:46164
I20240503 13:59:12.906293 5643 tablet_copy_client.cc:783] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:12.906477 5643 tablet_copy_client.cc:647] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:12.909082 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-c1c58189e2df41c3b8b4a0924ae42a03 received from {username='slave'} at 127.0.99.134:36473
I20240503 13:59:12.909184 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-c1c58189e2df41c3b8b4a0924ae42a03 on tablet c1c58189e2df41c3b8b4a0924ae42a03 with peer 6d1a97f876fc4f8698f737fcf1b782ad
I20240503 13:59:12.910243 5348 consensus_queue.cc:1035] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240503 13:59:12.911363 5348 consensus_queue.cc:1035] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 13:59:12.911671 3737 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index 25 to 51, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.912434 3870 consensus_queue.cc:237] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:12.914296 4973 raft_consensus.cc:1261] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240503 13:59:12.914389 5643 tablet_copy_client.cc:523] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.916257 5241 consensus_queue.cc:1035] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20240503 13:59:12.917562 5643 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap starting.
I20240503 13:59:12.918206 4622 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index 51 to 52, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.919152 4973 raft_consensus.cc:2932] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index 51 to 52, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:12.921116 3464 catalog_manager.cc:5484] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 51 to 52, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:12.924799 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 52) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:12.925550 3738 raft_consensus.cc:1261] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 13:59:12.927639 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62fbc3876d064aa0a8e7f8f71fb3f35f with cas_config_opid_index 51: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:12.928979 5537 tablet_copy_client.cc:523] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.929685 5348 consensus_queue.cc:1035] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 13:59:12.930882 5537 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap starting.
I20240503 13:59:12.934873 5367 raft_consensus.cc:1050] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d to VOTER
I20240503 13:59:12.935421 5367 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:12.937161 4844 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:12.937494 3606 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:12.937602 4756 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 13:59:12.938378 4756 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 13:59:12.940057 4756 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 46 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.940659 3606 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 46 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.940815 4844 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 46 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.943321 3464 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d reported cstate change: config changed from index 46 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.947139 3738 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:12.948175 4622 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index 45 to 71, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:12.949062 4844 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240503 13:59:12.950052 5367 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 13:59:12.950862 3459 catalog_manager.cc:5484] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 45 to 71, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:12.951943 4756 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.952389 4844 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.953349 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd23523f46044bb7b7e4f44056687c43 with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:12.954726 5242 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 45 to 71, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:12.961409 3464 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d reported cstate change: config changed from index 74 to 75, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:12.966625 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet bd23523f46044bb7b7e4f44056687c43 with delete_type TABLET_DATA_TOMBSTONED (TS bc41cca530f54259aca15e34563208dc not found in new config with opid_index 75) from {username='slave'} at 127.0.0.1:56666
I20240503 13:59:12.967530 5107 raft_consensus.cc:1203] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.55] Dedup: 1.50->[1.51-1.55]
I20240503 13:59:12.968678 5685 tablet_replica.cc:330] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: stopping tablet replica
I20240503 13:59:12.969019 5685 raft_consensus.cc:2229] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:12.969206 5685 raft_consensus.cc:2258] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:12.970777 5685 ts_tablet_manager.cc:1875] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:12.971588 5686 tablet_replica.cc:330] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:12.975215 5658 tablet_replica.cc:330] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: stopping tablet replica
I20240503 13:59:12.975598 5658 raft_consensus.cc:2229] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:12.975819 5658 raft_consensus.cc:2258] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:12.976073 5686 raft_consensus.cc:2229] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:12.976428 5686 raft_consensus.cc:2258] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:12.976797 5658 ts_tablet_manager.cc:1875] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:12.982600 5686 ts_tablet_manager.cc:1875] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:12.982633 5658 ts_tablet_manager.cc:1888] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240503 13:59:12.982857 5658 log.cc:1196] T bd23523f46044bb7b7e4f44056687c43 P bc41cca530f54259aca15e34563208dc: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:12.984053 3451 catalog_manager.cc:4830] TS bc41cca530f54259aca15e34563208dc (127.0.99.129:43281): tablet bd23523f46044bb7b7e4f44056687c43 (table rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]) successfully deleted
I20240503 13:59:13.003643 3737 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 45 to 71, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.004146 5685 ts_tablet_manager.cc:1888] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.75
I20240503 13:59:13.004256 5685 log.cc:1196] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:13.008576 5367 raft_consensus.cc:1050] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad to VOTER
I20240503 13:59:13.008949 5367 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.010239 5107 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 13:59:13.010671 4756 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 13:59:13.011406 5686 ts_tablet_manager.cc:1888] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.51
I20240503 13:59:13.012370 5490 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index 25 to 57, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.015240 3458 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 25 to 57, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.015311 5107 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index 25 to 57, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.019941 3738 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.021836 5107 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240503 13:59:13.022305 5367 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 13:59:13.026027 4756 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index 57 to 58, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.028839 3457 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 57 to 58, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.031199 3870 consensus_queue.cc:237] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:13.032831 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49afc417528049839897ee5600b954e8 with cas_config_opid_index 57: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.032943 5107 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 57 to 58, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.038507 5242 raft_consensus.cc:1203] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:13.039186 5686 log.cc:1196] T 62fbc3876d064aa0a8e7f8f71fb3f35f P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/62fbc3876d064aa0a8e7f8f71fb3f35f
I20240503 13:59:13.039265 5240 raft_consensus.cc:1261] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240503 13:59:13.039218 3452 catalog_manager.cc:4830] TS 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619): tablet 19ebb671e56f4e29adb6a2c0a7c6b00e (table rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]) successfully deleted
I20240503 13:59:13.041378 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet 62fbc3876d064aa0a8e7f8f71fb3f35f (table rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]) successfully deleted
I20240503 13:59:13.059129 3870 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 13:59:13.061868 5580 ts_tablet_manager.cc:902] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:13.062072 3849 tablet_service.cc:1514] Processing DeleteTablet for tablet 49afc417528049839897ee5600b954e8 with delete_type TABLET_DATA_TOMBSTONED (TS 0c6ca23f9d9b4c91a6347c0924674bdd not found in new config with opid_index 58) from {username='slave'} at 127.0.0.1:46164
I20240503 13:59:13.062641 5580 tablet_copy_client.cc:308] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:13.064317 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet e967e4257bda4e4fa9b3fdfadd22368f from peer 6d1a97f876fc4f8698f737fcf1b782ad ({username='slave'} at 127.0.99.134:36473)
I20240503 13:59:13.064443 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet e967e4257bda4e4fa9b3fdfadd22368f from peer 6d1a97f876fc4f8698f737fcf1b782ad at {username='slave'} at 127.0.99.134:36473: session id = 6d1a97f876fc4f8698f737fcf1b782ad-e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:13.065408 3759 tablet_copy_source_session.cc:205] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:13.066598 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e967e4257bda4e4fa9b3fdfadd22368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.068737 4622 consensus_queue.cc:1035] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240503 13:59:13.069388 5580 tablet_copy_client.cc:783] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:13.069597 5580 tablet_copy_client.cc:647] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:13.073786 5685 tablet_replica.cc:330] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: stopping tablet replica
I20240503 13:59:13.073952 5685 raft_consensus.cc:2229] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.074069 5685 pending_rounds.cc:62] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Trying to abort 1 pending ops.
I20240503 13:59:13.074144 5685 pending_rounds.cc:68] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Aborting op as it isn't in flight: id { term: 1 index: 56 } timestamp: 7023594508303622144 op_type: WRITE_OP write_request { tablet_id: "49afc417528049839897ee5600b954e8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\253kD4\345!\240\354\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""1f12ce4797499029" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023594508211826688 authz_token { token_data: "\010\332\335\323\261\006\"3\n\005slave\022*\n 1ad0716f24824080a4c3f0151947dfc9\020\001\030\001 \001(\001" signature: "<redacted>""\246(Yg;\203g\360+6\022Xb\032\036\251N\231F9\335oX@\260r\241DyM\212\243\036z\'\316\367X\233\277\235\007\264\"d\031>\3173\354\2561\233\250\206\033\260C#T\332\223\201\r" signing_key_seq_num: 0 } } request_id { client_id: "0f0f4b6faf8a46e19ca260d581153ae9" seq_no: 132 first_incomplete_seq_no: 132 attempt_no: 0 }
I20240503 13:59:13.074478 5685 raft_consensus.cc:2258] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.081455 5685 ts_tablet_manager.cc:1875] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.085742 4844 raft_consensus.cc:1203] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.73] Dedup: 1.66->[1.67-1.73]
I20240503 13:59:13.105090 4622 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.107909 5580 tablet_copy_client.cc:523] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:13.108242 3457 catalog_manager.cc:5484] T fc8c685a1dd4496b9be2b1e09d237371 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 71 to 73, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:13.109498 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc8c685a1dd4496b9be2b1e09d237371 with cas_config_opid_index 71: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.109536 5580 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap starting.
I20240503 13:59:13.110373 5240 raft_consensus.cc:2932] T fc8c685a1dd4496b9be2b1e09d237371 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.131389 5450 raft_consensus.cc:1050] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d to VOTER
I20240503 13:59:13.131824 5450 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:13.136523 3737 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:13.136816 4843 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:13.136967 5241 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 13:59:13.140308 4622 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 13:59:13.141782 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet fc8c685a1dd4496b9be2b1e09d237371 with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:13.143965 5686 tablet_replica.cc:330] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:13.144121 5686 raft_consensus.cc:2229] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.145320 5686 raft_consensus.cc:2258] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.148514 5348 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 55 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.152207 3458 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 55 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:13.155679 3737 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 55 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.161152 4843 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 55 to 74, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.167234 5686 ts_tablet_manager.cc:1875] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.171204 5686 ts_tablet_manager.cc:1888] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72
I20240503 13:59:13.171353 5686 log.cc:1196] T fc8c685a1dd4496b9be2b1e09d237371 P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fc8c685a1dd4496b9be2b1e09d237371
I20240503 13:59:13.175487 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet fc8c685a1dd4496b9be2b1e09d237371 (table rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]) successfully deleted
I20240503 13:59:13.188966 3870 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:13.190383 4844 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240503 13:59:13.192566 5377 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 13:59:13.198904 5241 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.202296 3458 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 74 to 75, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.206358 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet e572db34c8844384b728068168645ecc with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 75) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:13.208214 5686 tablet_replica.cc:330] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:13.208482 5686 raft_consensus.cc:2229] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.208797 5686 raft_consensus.cc:2258] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.209656 5686 ts_tablet_manager.cc:1875] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.210543 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e572db34c8844384b728068168645ecc with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.211230 4844 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.212576 5686 ts_tablet_manager.cc:1888] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240503 13:59:13.213546 5686 log.cc:1196] T e572db34c8844384b728068168645ecc P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e572db34c8844384b728068168645ecc
I20240503 13:59:13.222018 5537 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:13.222035 5685 ts_tablet_manager.cc:1888] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.56
I20240503 13:59:13.222142 5685 log.cc:1196] T 49afc417528049839897ee5600b954e8 P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/49afc417528049839897ee5600b954e8
I20240503 13:59:13.222553 5537 tablet_bootstrap.cc:492] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: Bootstrap complete.
I20240503 13:59:13.223965 3452 catalog_manager.cc:4830] TS 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619): tablet 49afc417528049839897ee5600b954e8 (table rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]) successfully deleted
I20240503 13:59:13.224319 5537 ts_tablet_manager.cc:1367] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: Time spent bootstrapping tablet: real 0.293s user 0.075s sys 0.008s
I20240503 13:59:13.224885 5537 raft_consensus.cc:348] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:13.232101 5537 raft_consensus.cc:729] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e553631c0564b078cefd1cd1cdcc90d, State: Initialized, Role: LEARNER
I20240503 13:59:13.232426 5537 consensus_queue.cc:260] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: true } }
I20240503 13:59:13.234969 5537 ts_tablet_manager.cc:1398] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 13:59:13.242285 3628 tablet_copy_service.cc:342] P bc41cca530f54259aca15e34563208dc: Request end of tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-adfa836cb94c4050bdb65e007b232f2b received from {username='slave'} at 127.0.99.132:60531
I20240503 13:59:13.242421 3628 tablet_copy_service.cc:433] P bc41cca530f54259aca15e34563208dc: ending tablet copy session 4e553631c0564b078cefd1cd1cdcc90d-adfa836cb94c4050bdb65e007b232f2b on tablet adfa836cb94c4050bdb65e007b232f2b with peer 4e553631c0564b078cefd1cd1cdcc90d
I20240503 13:59:13.232688 5556 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=87 ignored=0} inserts{seen=718 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:13.245671 5556 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: Bootstrap complete.
I20240503 13:59:13.245919 5556 ts_tablet_manager.cc:1367] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: Time spent bootstrapping tablet: real 0.466s user 0.088s sys 0.004s
I20240503 13:59:13.246871 5543 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=710 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:13.247124 5543 tablet_bootstrap.cc:492] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap complete.
I20240503 13:59:13.247294 5543 ts_tablet_manager.cc:1367] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent bootstrapping tablet: real 0.365s user 0.075s sys 0.008s
I20240503 13:59:13.247705 5543 raft_consensus.cc:348] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:13.247824 5543 raft_consensus.cc:729] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14a79ecc8c2c45adac20a5239cd45fd2, State: Initialized, Role: LEARNER
I20240503 13:59:13.247920 5543 consensus_queue.cc:260] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:13.246438 5556 raft_consensus.cc:348] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:13.248142 5556 raft_consensus.cc:729] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07bdad76c7bd40afb3147833b3305924, State: Initialized, Role: LEARNER
I20240503 13:59:13.248255 5556 consensus_queue.cc:260] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:13.248416 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet e572db34c8844384b728068168645ecc (table rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]) successfully deleted
I20240503 13:59:13.250190 5556 ts_tablet_manager.cc:1398] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:13.251042 5543 ts_tablet_manager.cc:1398] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:13.251502 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-809441335aa64f959ef084cb3c8b2434 received from {username='slave'} at 127.0.99.133:48411
I20240503 13:59:13.251586 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-809441335aa64f959ef084cb3c8b2434 on tablet 809441335aa64f959ef084cb3c8b2434 with peer 14a79ecc8c2c45adac20a5239cd45fd2
I20240503 13:59:13.253201 3628 tablet_copy_service.cc:342] P bc41cca530f54259aca15e34563208dc: Request end of tablet copy session 07bdad76c7bd40afb3147833b3305924-97d59e7bd0da44d9a7af791f2477b183 received from {username='slave'} at 127.0.99.135:48765
I20240503 13:59:13.253283 3628 tablet_copy_service.cc:433] P bc41cca530f54259aca15e34563208dc: ending tablet copy session 07bdad76c7bd40afb3147833b3305924-97d59e7bd0da44d9a7af791f2477b183 on tablet 97d59e7bd0da44d9a7af791f2477b183 with peer 07bdad76c7bd40afb3147833b3305924
I20240503 13:59:13.329351 5580 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:13.329675 5580 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap complete.
I20240503 13:59:13.329875 5580 ts_tablet_manager.cc:1367] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent bootstrapping tablet: real 0.220s user 0.065s sys 0.021s
I20240503 13:59:13.331794 5580 raft_consensus.cc:348] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:13.331950 5580 raft_consensus.cc:729] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Initialized, Role: LEARNER
I20240503 13:59:13.332047 5580 consensus_queue.cc:260] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:13.332969 5580 ts_tablet_manager.cc:1398] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:13.333505 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-e967e4257bda4e4fa9b3fdfadd22368f received from {username='slave'} at 127.0.99.134:36473
I20240503 13:59:13.333587 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-e967e4257bda4e4fa9b3fdfadd22368f on tablet e967e4257bda4e4fa9b3fdfadd22368f with peer 6d1a97f876fc4f8698f737fcf1b782ad
I20240503 13:59:13.343675 5107 raft_consensus.cc:1203] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.91] Dedup: 1.70->[1.71-1.91]
I20240503 13:59:13.365696 5367 raft_consensus.cc:1050] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad to VOTER
I20240503 13:59:13.366101 5367 consensus_queue.cc:237] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.367312 5107 raft_consensus.cc:1261] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240503 13:59:13.367657 4756 consensus_queue.cc:1035] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240503 13:59:13.369956 5490 raft_consensus.cc:2932] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index 55 to 92, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.370348 5106 raft_consensus.cc:2932] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index 55 to 92, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.373452 3458 catalog_manager.cc:5484] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad reported cstate change: config changed from index 55 to 92, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.378505 3737 consensus_queue.cc:237] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.379968 5106 raft_consensus.cc:1261] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240503 13:59:13.380661 4756 consensus_queue.cc:1035] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 13:59:13.384523 3604 raft_consensus.cc:1261] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240503 13:59:13.393072 5380 raft_consensus.cc:2932] T c1c58189e2df41c3b8b4a0924ae42a03 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 92 to 93, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.393877 5106 raft_consensus.cc:2932] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 92 to 93, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.413250 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c1c58189e2df41c3b8b4a0924ae42a03 with cas_config_opid_index 92: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.414023 3457 catalog_manager.cc:5484] T c1c58189e2df41c3b8b4a0924ae42a03 P 6d1a97f876fc4f8698f737fcf1b782ad reported cstate change: config changed from index 92 to 93, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.416635 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c58189e2df41c3b8b4a0924ae42a03 with delete_type TABLET_DATA_TOMBSTONED (TS bc41cca530f54259aca15e34563208dc not found in new config with opid_index 93) from {username='slave'} at 127.0.0.1:56666
I20240503 13:59:13.416868 5658 tablet_replica.cc:330] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: stopping tablet replica
I20240503 13:59:13.417017 5658 raft_consensus.cc:2229] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.417383 5658 raft_consensus.cc:2258] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.418658 5658 ts_tablet_manager.cc:1875] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.421914 5658 ts_tablet_manager.cc:1888] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.91
I20240503 13:59:13.422032 5658 log.cc:1196] T c1c58189e2df41c3b8b4a0924ae42a03 P bc41cca530f54259aca15e34563208dc: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c1c58189e2df41c3b8b4a0924ae42a03
I20240503 13:59:13.425355 3451 catalog_manager.cc:4830] TS bc41cca530f54259aca15e34563208dc (127.0.99.129:43281): tablet c1c58189e2df41c3b8b4a0924ae42a03 (table rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]) successfully deleted
I20240503 13:59:13.438427 4843 raft_consensus.cc:1203] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.92] Dedup: 1.79->[1.80-1.92]
I20240503 13:59:13.464520 5450 raft_consensus.cc:1050] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 07bdad76c7bd40afb3147833b3305924 to VOTER
I20240503 13:59:13.464928 5450 consensus_queue.cc:237] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:13.466734 5105 raft_consensus.cc:1261] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:13.468360 4622 consensus_queue.cc:1035] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:13.469439 5243 raft_consensus.cc:1261] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:13.470744 5377 consensus_queue.cc:1035] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:13.473330 4490 raft_consensus.cc:1050] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc: attempting to promote NON_VOTER 4e553631c0564b078cefd1cd1cdcc90d to VOTER
I20240503 13:59:13.473793 4490 consensus_queue.cc:237] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:13.476687 4843 raft_consensus.cc:1261] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 LEARNER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 13:59:13.477088 5318 consensus_queue.cc:1035] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240503 13:59:13.477756 5348 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.478518 5240 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.480554 3869 raft_consensus.cc:1261] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 13:59:13.481554 3457 catalog_manager.cc:5484] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: config changed from index 2 to 3, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.482084 5105 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.483080 5318 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index 74 to 95, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.486873 4843 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 74 to 95, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.490588 3457 catalog_manager.cc:5484] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d reported cstate change: config changed from index 74 to 95, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.492718 5318 consensus_queue.cc:1035] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240503 13:59:13.494097 5643 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:13.494025 3869 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 74 to 95, 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.494433 5643 tablet_bootstrap.cc:492] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: Bootstrap complete.
I20240503 13:59:13.494671 5643 ts_tablet_manager.cc:1367] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent bootstrapping tablet: real 0.577s user 0.068s sys 0.008s
I20240503 13:59:13.495152 5643 raft_consensus.cc:348] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:13.495270 5643 raft_consensus.cc:729] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Initialized, Role: LEARNER
I20240503 13:59:13.495368 5643 consensus_queue.cc:260] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: true } }
I20240503 13:59:13.496109 5643 ts_tablet_manager.cc:1398] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:13.498961 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-dcf42adb4ddc4a25907a5c5fa94e31b7 received from {username='slave'} at 127.0.99.134:40103
I20240503 13:59:13.499056 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 6d1a97f876fc4f8698f737fcf1b782ad-dcf42adb4ddc4a25907a5c5fa94e31b7 on tablet dcf42adb4ddc4a25907a5c5fa94e31b7 with peer 6d1a97f876fc4f8698f737fcf1b782ad
I20240503 13:59:13.505868 3607 consensus_queue.cc:237] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } }
I20240503 13:59:13.508385 4843 raft_consensus.cc:1261] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240503 13:59:13.509531 5384 consensus_queue.cc:1035] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 13:59:13.512014 5384 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 95 to 96, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.514708 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adfa836cb94c4050bdb65e007b232f2b with cas_config_opid_index 95: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.516413 4843 raft_consensus.cc:2932] T adfa836cb94c4050bdb65e007b232f2b P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 95 to 96, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } }
I20240503 13:59:13.519027 3464 catalog_manager.cc:5484] T adfa836cb94c4050bdb65e007b232f2b P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 95 to 96, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.523078 3849 tablet_service.cc:1514] Processing DeleteTablet for tablet adfa836cb94c4050bdb65e007b232f2b with delete_type TABLET_DATA_TOMBSTONED (TS 0c6ca23f9d9b4c91a6347c0924674bdd not found in new config with opid_index 96) from {username='slave'} at 127.0.0.1:46164
I20240503 13:59:13.523320 5685 tablet_replica.cc:330] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: stopping tablet replica
I20240503 13:59:13.523470 5685 raft_consensus.cc:2229] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.523730 5685 raft_consensus.cc:2258] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.524873 5685 ts_tablet_manager.cc:1875] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.527313 5685 ts_tablet_manager.cc:1888] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.95
I20240503 13:59:13.527386 5685 log.cc:1196] T adfa836cb94c4050bdb65e007b232f2b P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/adfa836cb94c4050bdb65e007b232f2b
I20240503 13:59:13.532593 3452 catalog_manager.cc:4830] TS 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619): tablet adfa836cb94c4050bdb65e007b232f2b (table rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]) successfully deleted
I20240503 13:59:13.546231 5107 raft_consensus.cc:1203] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.80->[1.81-1.104] Dedup: 1.92->[1.93-1.104]
I20240503 13:59:13.587455 5240 raft_consensus.cc:1203] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.68->[1.69-1.108] Dedup: 1.87->[1.88-1.108]
I20240503 13:59:13.621942 5380 raft_consensus.cc:1050] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad to VOTER
I20240503 13:59:13.622349 5380 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.623528 3603 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 13:59:13.625000 4756 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 13:59:13.627110 5106 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 13:59:13.628362 5327 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 13:59:13.628419 5367 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 81 to 108, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.631464 3464 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 81 to 108, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:13.631693 5107 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 81 to 108, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.635849 3604 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 81 to 108, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.649883 4974 raft_consensus.cc:1203] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.62->[1.63-1.95] Dedup: 1.79->[1.80-1.95]
I20240503 13:59:13.651675 3737 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.656971 4490 raft_consensus.cc:1050] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: attempting to promote NON_VOTER 07bdad76c7bd40afb3147833b3305924 to VOTER
I20240503 13:59:13.657435 4490 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:13.657581 5107 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 13:59:13.658337 5380 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240503 13:59:13.658826 5240 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 13:59:13.659837 5679 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240503 13:59:13.660635 3737 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 13:59:13.663511 5318 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 69 to 110, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.664857 5379 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 108 to 110, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.665923 5679 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240503 13:59:13.666676 5240 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 69 to 110, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.667788 3464 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 69 to 110, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.669996 3457 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 108 to 110, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.675189 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet e967e4257bda4e4fa9b3fdfadd22368f with delete_type TABLET_DATA_TOMBSTONED (TS bc41cca530f54259aca15e34563208dc not found in new config with opid_index 110) from {username='slave'} at 127.0.0.1:56666
I20240503 13:59:13.675442 5658 tablet_replica.cc:330] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: stopping tablet replica
I20240503 13:59:13.675595 5658 raft_consensus.cc:2229] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.675643 3604 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:13.675776 5658 raft_consensus.cc:2258] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.676651 5658 ts_tablet_manager.cc:1875] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.677703 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e967e4257bda4e4fa9b3fdfadd22368f with cas_config_opid_index 108: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.678085 5240 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240503 13:59:13.678568 4490 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240503 13:59:13.679216 5658 ts_tablet_manager.cc:1888] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.109
I20240503 13:59:13.679308 5658 log.cc:1196] T e967e4257bda4e4fa9b3fdfadd22368f P bc41cca530f54259aca15e34563208dc: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:13.679364 5107 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 108 to 110, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.679945 3451 catalog_manager.cc:4830] TS bc41cca530f54259aca15e34563208dc (127.0.99.129:43281): tablet e967e4257bda4e4fa9b3fdfadd22368f (table rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]) successfully deleted
I20240503 13:59:13.680840 4490 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 110 to 111, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.682348 3737 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 69 to 110, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.684185 5106 raft_consensus.cc:1203] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.98] Dedup: 1.71->[1.72-1.98]
I20240503 13:59:13.689572 5240 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 110 to 111, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:13.693348 3464 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 110 to 111, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.694612 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97d59e7bd0da44d9a7af791f2477b183 with cas_config_opid_index 110: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.697113 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d59e7bd0da44d9a7af791f2477b183 with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 111) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:13.697351 5686 tablet_replica.cc:330] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:13.697505 5686 raft_consensus.cc:2229] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.697906 5686 raft_consensus.cc:2258] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.701550 5686 ts_tablet_manager.cc:1875] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.704177 5686 ts_tablet_manager.cc:1888] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.110
I20240503 13:59:13.704289 5686 log.cc:1196] T 97d59e7bd0da44d9a7af791f2477b183 P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:13.708575 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet 97d59e7bd0da44d9a7af791f2477b183 (table rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]) successfully deleted
I20240503 13:59:13.742409 5241 raft_consensus.cc:1050] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 6d1a97f876fc4f8698f737fcf1b782ad to VOTER
I20240503 13:59:13.742734 5241 consensus_queue.cc:237] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.100, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.744138 3607 raft_consensus.cc:1261] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240503 13:59:13.745005 5450 consensus_queue.cc:1035] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 99, Time since last communication: 0.000s
I20240503 13:59:13.750432 5241 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.101: config changed from index 62 to 101, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.752563 5106 raft_consensus.cc:1261] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240503 13:59:13.752734 3607 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 62 to 101, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.753062 5241 consensus_queue.cc:1035] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 99, Time since last communication: 0.000s
I20240503 13:59:13.755210 4622 raft_consensus.cc:1050] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 to VOTER
I20240503 13:59:13.755558 4622 consensus_queue.cc:237] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:13.755559 3464 catalog_manager.cc:5484] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 62 to 101, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.754853 5107 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 62 to 101, 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.758229 3607 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 13:59:13.758611 4974 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 13:59:13.758661 5241 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 13:59:13.759325 5241 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 13:59:13.761528 4622 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 63 to 97, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:13.762625 4974 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 63 to 97, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:13.765440 3458 catalog_manager.cc:5484] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 63 to 97, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:13.768733 3607 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 63 to 97, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:13.774605 3869 consensus_queue.cc:237] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } }
I20240503 13:59:13.778853 5106 raft_consensus.cc:1261] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 13:59:13.779389 5348 consensus_queue.cc:1035] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 13:59:13.783015 5377 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 101 to 103, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.783336 5106 raft_consensus.cc:2932] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 101 to 103, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.785248 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dcf42adb4ddc4a25907a5c5fa94e31b7 with cas_config_opid_index 101: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.787281 3457 catalog_manager.cc:5484] T dcf42adb4ddc4a25907a5c5fa94e31b7 P 6d1a97f876fc4f8698f737fcf1b782ad reported cstate change: config changed from index 101 to 103, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } }
I20240503 13:59:13.791986 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf42adb4ddc4a25907a5c5fa94e31b7 with delete_type TABLET_DATA_TOMBSTONED (TS bc41cca530f54259aca15e34563208dc not found in new config with opid_index 103) from {username='slave'} at 127.0.0.1:56666
I20240503 13:59:13.792230 5658 tablet_replica.cc:330] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: stopping tablet replica
I20240503 13:59:13.792385 5658 raft_consensus.cc:2229] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.792546 5658 raft_consensus.cc:2258] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.793632 5658 ts_tablet_manager.cc:1875] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.796241 5658 ts_tablet_manager.cc:1888] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20240503 13:59:13.796332 5658 log.cc:1196] T dcf42adb4ddc4a25907a5c5fa94e31b7 P bc41cca530f54259aca15e34563208dc: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/dcf42adb4ddc4a25907a5c5fa94e31b7
I20240503 13:59:13.796844 3451 catalog_manager.cc:4830] TS bc41cca530f54259aca15e34563208dc (127.0.99.129:43281): tablet dcf42adb4ddc4a25907a5c5fa94e31b7 (table rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]) successfully deleted
I20240503 13:59:13.919116 3869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434"
dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57264
I20240503 13:59:13.919343 3869 raft_consensus.cc:595] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:13.994724 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.001235 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
I20240503 13:59:14.002673 3869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8df17b9134914ca3a2c04800435b3697"
dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57298
I20240503 13:59:14.002950 3869 raft_consensus.cc:595] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:14.008227 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.013620 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.019625 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.030800 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.039508 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.049505 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
I20240503 13:59:14.055130 4622 raft_consensus.cc:979] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: : Instructing follower 6d1a97f876fc4f8698f737fcf1b782ad to start an election
I20240503 13:59:14.055274 4622 raft_consensus.cc:1067] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Signalling peer 6d1a97f876fc4f8698f737fcf1b782ad to start an election
I20240503 13:59:14.055841 5104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8df17b9134914ca3a2c04800435b3697"
dest_uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
from {username='slave'} at 127.0.99.131:60701
I20240503 13:59:14.056038 5104 raft_consensus.cc:482] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:14.056146 5104 raft_consensus.cc:3037] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:14.057688 5104 raft_consensus.cc:504] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:14.059438 5104 leader_election.cc:288] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [CANDIDATE]: Term 2 election: Requested vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619), 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643)
I20240503 13:59:14.062086 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df17b9134914ca3a2c04800435b3697" candidate_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
I20240503 13:59:14.062227 3869 raft_consensus.cc:3032] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:14.062268 3869 raft_consensus.cc:729] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:14.062443 3869 consensus_queue.cc:260] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:14.062605 3869 raft_consensus.cc:3037] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:14.063663 3869 raft_consensus.cc:2454] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1a97f876fc4f8698f737fcf1b782ad in term 2.
I20240503 13:59:14.064415 5041 leader_election.cc:302] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 6d1a97f876fc4f8698f737fcf1b782ad; no voters:
I20240503 13:59:14.064707 5677 raft_consensus.cc:2781] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:14.064885 5677 raft_consensus.cc:686] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 LEADER]: Becoming Leader. State: Replica: 6d1a97f876fc4f8698f737fcf1b782ad, State: Running, Role: LEADER
I20240503 13:59:14.065217 5677 consensus_queue.cc:237] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
W20240503 13:59:14.062270 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
I20240503 13:59:14.068207 3458 catalog_manager.cc:5484] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad reported cstate change: term changed from 1 to 2, leader changed from 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) to 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134). New cstate: current_term: 2 leader_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:14.075946 3869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434"
dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57330
I20240503 13:59:14.076103 3869 raft_consensus.cc:595] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:14.079391 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.091989 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.104739 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
I20240503 13:59:14.111598 5240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df17b9134914ca3a2c04800435b3697" candidate_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07bdad76c7bd40afb3147833b3305924"
I20240503 13:59:14.111871 5240 raft_consensus.cc:3037] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:14.113225 5240 raft_consensus.cc:2454] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1a97f876fc4f8698f737fcf1b782ad in term 2.
W20240503 13:59:14.122690 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.141104 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.160562 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.180970 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.200888 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.219096 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.238843 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
W20240503 13:59:14.268878 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
I20240503 13:59:14.287943 4622 raft_consensus.cc:979] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: : Instructing follower bc41cca530f54259aca15e34563208dc to start an election
I20240503 13:59:14.288146 5377 raft_consensus.cc:1067] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Signalling peer bc41cca530f54259aca15e34563208dc to start an election
I20240503 13:59:14.288596 3603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434"
dest_uuid: "bc41cca530f54259aca15e34563208dc"
from {username='slave'} at 127.0.99.131:44985
I20240503 13:59:14.288784 3603 raft_consensus.cc:482] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:14.288851 3603 raft_consensus.cc:3037] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:14.291888 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Service unavailable: leader transfer in progress
I20240503 13:59:14.294116 3603 raft_consensus.cc:504] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:14.295017 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434" candidate_uuid: "bc41cca530f54259aca15e34563208dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 103 } } ignore_live_leader: true dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
I20240503 13:59:14.295173 3869 raft_consensus.cc:3032] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:14.295228 3869 raft_consensus.cc:729] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:14.295388 3869 consensus_queue.cc:260] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:14.295571 3869 raft_consensus.cc:3037] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:14.295564 3603 leader_election.cc:288] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 2 election: Requested vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619), 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785)
I20240503 13:59:14.296686 3869 raft_consensus.cc:2454] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc41cca530f54259aca15e34563208dc in term 2.
I20240503 13:59:14.298831 3543 leader_election.cc:302] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, bc41cca530f54259aca15e34563208dc; no voters:
I20240503 13:59:14.299012 4490 raft_consensus.cc:2781] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:14.299084 4490 raft_consensus.cc:686] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 2 LEADER]: Becoming Leader. State: Replica: bc41cca530f54259aca15e34563208dc, State: Running, Role: LEADER
I20240503 13:59:14.299232 4490 consensus_queue.cc:237] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:14.301424 3458 catalog_manager.cc:5484] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc reported cstate change: term changed from 1 to 2, leader changed from 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) to bc41cca530f54259aca15e34563208dc (127.0.99.129). New cstate: current_term: 2 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:14.317687 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57126: Illegal state: replica 0c6ca23f9d9b4c91a6347c0924674bdd is not leader of this config: current role FOLLOWER
I20240503 13:59:14.347244 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809441335aa64f959ef084cb3c8b2434" candidate_uuid: "bc41cca530f54259aca15e34563208dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 103 } } ignore_live_leader: true dest_uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
I20240503 13:59:14.348372 4975 raft_consensus.cc:3037] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:14.350113 4975 raft_consensus.cc:2454] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc41cca530f54259aca15e34563208dc in term 2.
I20240503 13:59:14.378340 4975 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 2 index: 105. (index mismatch)
I20240503 13:59:14.379235 3869 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 2 index: 105. (index mismatch)
I20240503 13:59:14.379947 5318 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 13:59:14.380745 5851 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 13:59:14.389341 5809 mvcc.cc:205] Tried to move back new op lower bound from 7023594513928265728 to 7023594513611251712. Current Snapshot: MvccSnapshot[applied={T|T < 7023594513928265728 or (T in {7023594513928265728})}]
I20240503 13:59:14.399030 3604 consensus_queue.cc:237] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 2.105, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:14.403366 4975 raft_consensus.cc:1261] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch)
I20240503 13:59:14.406595 5851 consensus_queue.cc:1035] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240503 13:59:14.418434 5318 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P bc41cca530f54259aca15e34563208dc [term 2 LEADER]: Committing config change with OpId 2.106: config changed from index 97 to 106, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:14.420071 4973 raft_consensus.cc:2932] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Committing config change with OpId 2.106: config changed from index 97 to 106, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:14.426326 3458 catalog_manager.cc:5484] T 809441335aa64f959ef084cb3c8b2434 P 14a79ecc8c2c45adac20a5239cd45fd2 reported cstate change: config changed from index 97 to 106, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:14.428416 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 809441335aa64f959ef084cb3c8b2434 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:14.430997 3849 tablet_service.cc:1514] Processing DeleteTablet for tablet 809441335aa64f959ef084cb3c8b2434 with delete_type TABLET_DATA_TOMBSTONED (TS 0c6ca23f9d9b4c91a6347c0924674bdd not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:46164
I20240503 13:59:14.434368 5854 tablet_replica.cc:330] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: stopping tablet replica
I20240503 13:59:14.435039 5854 raft_consensus.cc:2229] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:14.435343 5854 raft_consensus.cc:2258] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:14.444937 5854 ts_tablet_manager.cc:1875] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:14.455619 5243 raft_consensus.cc:1261] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Refusing update from remote peer 6d1a97f876fc4f8698f737fcf1b782ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:14.451777 5854 ts_tablet_manager.cc:1888] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.105
I20240503 13:59:14.456175 5854 log.cc:1196] T 809441335aa64f959ef084cb3c8b2434 P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/809441335aa64f959ef084cb3c8b2434
I20240503 13:59:14.456135 5828 consensus_queue.cc:1035] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:14.457870 3452 catalog_manager.cc:4830] TS 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619): tablet 809441335aa64f959ef084cb3c8b2434 (table rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]) successfully deleted
I20240503 13:59:14.464479 3869 raft_consensus.cc:1261] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Refusing update from remote peer 6d1a97f876fc4f8698f737fcf1b782ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:14.468014 5677 consensus_queue.cc:1035] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:14.493463 5105 consensus_queue.cc:237] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:14.511198 5243 raft_consensus.cc:1261] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Refusing update from remote peer 6d1a97f876fc4f8698f737fcf1b782ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:14.515292 5677 consensus_queue.cc:1035] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:14.528458 5828 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:14.530144 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8df17b9134914ca3a2c04800435b3697 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:14.531244 3458 catalog_manager.cc:5484] T 8df17b9134914ca3a2c04800435b3697 P 6d1a97f876fc4f8698f737fcf1b782ad reported cstate change: config changed from index 3 to 5, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:14.532898 5240 raft_consensus.cc:2932] T 8df17b9134914ca3a2c04800435b3697 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:14.534729 3849 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df17b9134914ca3a2c04800435b3697 with delete_type TABLET_DATA_TOMBSTONED (TS 0c6ca23f9d9b4c91a6347c0924674bdd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46164
I20240503 13:59:14.536058 5854 tablet_replica.cc:330] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: stopping tablet replica
I20240503 13:59:14.536262 5854 raft_consensus.cc:2229] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:14.536442 5854 raft_consensus.cc:2258] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:14.548321 5854 ts_tablet_manager.cc:1875] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:14.567623 5854 ts_tablet_manager.cc:1888] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:14.567736 5854 log.cc:1196] T 8df17b9134914ca3a2c04800435b3697 P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8df17b9134914ca3a2c04800435b3697
I20240503 13:59:14.571326 3452 catalog_manager.cc:4830] TS 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619): tablet 8df17b9134914ca3a2c04800435b3697 (table TestTable [id=69946089d0414709a39ce318b85d074e]) successfully deleted
I20240503 13:59:14.963006 3849 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:14.965087 3717 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:14.965276 4823 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:14.972133 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:14.981683 4955 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:14.994916 5217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:15.017841 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:15.168560 3737 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:15.170220 4842 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 13:59:15.170773 5490 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240503 13:59:15.172796 5490 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 75 to 123, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:15.175191 4843 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 75 to 123, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:15.179199 3458 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 75 to 123, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:15.182142 3675 consensus_peers.cc:485] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 -> Peer 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785): Couldn't send request to peer 14a79ecc8c2c45adac20a5239cd45fd2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd23523f46044bb7b7e4f44056687c43. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:15.228238 3869 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
W20240503 13:59:15.229735 3806 consensus_peers.cc:485] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd -> Peer 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785): Couldn't send request to peer 14a79ecc8c2c45adac20a5239cd45fd2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e572db34c8844384b728068168645ecc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:15.232064 4842 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240503 13:59:15.232594 5348 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 13:59:15.239602 4622 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.150: config changed from index 75 to 150, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:15.240370 4844 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index 75 to 150, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } } }
I20240503 13:59:15.244922 3464 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 75 to 150, NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:15.271209 3737 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:15.281009 5105 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 13:59:15.283123 5367 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 13:59:15.285928 5490 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index 58 to 127, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
I20240503 13:59:15.292263 3458 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 58 to 127, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:15.293952 5105 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 58 to 127, NON_VOTER 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } } }
W20240503 13:59:15.318639 3672 consensus_peers.cc:485] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 -> Peer 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643): Couldn't send request to peer 07bdad76c7bd40afb3147833b3305924. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49afc417528049839897ee5600b954e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:15.620417 5956 ts_tablet_manager.cc:902] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: Initiating tablet copy from peer 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619)
I20240503 13:59:15.621049 5956 tablet_copy_client.cc:308] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41619
I20240503 13:59:15.622260 3890 tablet_copy_service.cc:140] P 0c6ca23f9d9b4c91a6347c0924674bdd: Received BeginTabletCopySession request for tablet e572db34c8844384b728068168645ecc from peer 14a79ecc8c2c45adac20a5239cd45fd2 ({username='slave'} at 127.0.99.133:48411)
I20240503 13:59:15.622370 3890 tablet_copy_service.cc:161] P 0c6ca23f9d9b4c91a6347c0924674bdd: Beginning new tablet copy session on tablet e572db34c8844384b728068168645ecc from peer 14a79ecc8c2c45adac20a5239cd45fd2 at {username='slave'} at 127.0.99.133:48411: session id = 14a79ecc8c2c45adac20a5239cd45fd2-e572db34c8844384b728068168645ecc
I20240503 13:59:15.623301 3890 tablet_copy_source_session.cc:205] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:15.630784 5956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e572db34c8844384b728068168645ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.635696 5956 tablet_copy_client.cc:783] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:15.635946 5956 tablet_copy_client.cc:647] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:15.645918 5966 ts_tablet_manager.cc:902] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:15.652004 5966 tablet_copy_client.cc:308] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:15.652441 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet bd23523f46044bb7b7e4f44056687c43 from peer 14a79ecc8c2c45adac20a5239cd45fd2 ({username='slave'} at 127.0.99.133:34067)
I20240503 13:59:15.652546 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet bd23523f46044bb7b7e4f44056687c43 from peer 14a79ecc8c2c45adac20a5239cd45fd2 at {username='slave'} at 127.0.99.133:34067: session id = 14a79ecc8c2c45adac20a5239cd45fd2-bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:15.649963 5956 tablet_copy_client.cc:523] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:15.653420 3759 tablet_copy_source_session.cc:205] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:15.656080 5956 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap starting.
I20240503 13:59:15.657752 5966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd23523f46044bb7b7e4f44056687c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.660847 5966 tablet_copy_client.cc:783] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:15.661020 5966 tablet_copy_client.cc:647] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:15.664942 5966 tablet_copy_client.cc:523] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:15.673200 5976 ts_tablet_manager.cc:902] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:15.673897 5976 tablet_copy_client.cc:308] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:15.678404 5966 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap starting.
I20240503 13:59:15.684756 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet 49afc417528049839897ee5600b954e8 from peer 07bdad76c7bd40afb3147833b3305924 ({username='slave'} at 127.0.99.135:49075)
I20240503 13:59:15.684881 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet 49afc417528049839897ee5600b954e8 from peer 07bdad76c7bd40afb3147833b3305924 at {username='slave'} at 127.0.99.135:49075: session id = 07bdad76c7bd40afb3147833b3305924-49afc417528049839897ee5600b954e8
I20240503 13:59:15.685861 3759 tablet_copy_source_session.cc:205] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:15.686758 5976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49afc417528049839897ee5600b954e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:15.689256 5976 tablet_copy_client.cc:783] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:15.689404 5976 tablet_copy_client.cc:647] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:15.699659 5976 tablet_copy_client.cc:523] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:15.703619 5976 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Bootstrap starting.
I20240503 13:59:15.899430 5966 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=137 overwritten=0 applied=137 ignored=0} inserts{seen=819 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:15.900386 5966 tablet_bootstrap.cc:492] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap complete.
I20240503 13:59:15.900735 5966 ts_tablet_manager.cc:1367] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent bootstrapping tablet: real 0.222s user 0.114s sys 0.004s
I20240503 13:59:15.901393 5966 raft_consensus.cc:348] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:15.901660 5966 raft_consensus.cc:729] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14a79ecc8c2c45adac20a5239cd45fd2, State: Initialized, Role: LEARNER
I20240503 13:59:15.901827 5966 consensus_queue.cc:260] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 137, Last appended: 1.137, Last appended by leader: 137, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:15.902395 5966 ts_tablet_manager.cc:1398] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.902971 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-bd23523f46044bb7b7e4f44056687c43 received from {username='slave'} at 127.0.99.133:34067
I20240503 13:59:15.903059 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-bd23523f46044bb7b7e4f44056687c43 on tablet bd23523f46044bb7b7e4f44056687c43 with peer 14a79ecc8c2c45adac20a5239cd45fd2
I20240503 13:59:15.922205 5976 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Bootstrap replayed 1/1 log segments. Stats: ops{read=146 overwritten=0 applied=146 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:15.922713 5976 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Bootstrap complete.
I20240503 13:59:15.923003 5976 ts_tablet_manager.cc:1367] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Time spent bootstrapping tablet: real 0.219s user 0.109s sys 0.014s
I20240503 13:59:15.923566 5976 raft_consensus.cc:348] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:15.923790 5976 raft_consensus.cc:729] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07bdad76c7bd40afb3147833b3305924, State: Initialized, Role: LEARNER
I20240503 13:59:15.923951 5976 consensus_queue.cc:260] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 146, Last appended: 1.146, Last appended by leader: 146, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: true } }
I20240503 13:59:15.924413 5976 ts_tablet_manager.cc:1398] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:15.925418 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 07bdad76c7bd40afb3147833b3305924-49afc417528049839897ee5600b954e8 received from {username='slave'} at 127.0.99.135:49075
I20240503 13:59:15.925503 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 07bdad76c7bd40afb3147833b3305924-49afc417528049839897ee5600b954e8 on tablet 49afc417528049839897ee5600b954e8 with peer 07bdad76c7bd40afb3147833b3305924
I20240503 13:59:16.040409 5956 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=157 overwritten=0 applied=157 ignored=0} inserts{seen=785 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:16.040798 5956 tablet_bootstrap.cc:492] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: Bootstrap complete.
I20240503 13:59:16.041031 5956 ts_tablet_manager.cc:1367] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent bootstrapping tablet: real 0.385s user 0.132s sys 0.000s
I20240503 13:59:16.041517 5956 raft_consensus.cc:348] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:16.041743 5956 raft_consensus.cc:729] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14a79ecc8c2c45adac20a5239cd45fd2, State: Initialized, Role: LEARNER
I20240503 13:59:16.041880 5956 consensus_queue.cc:260] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 157, Last appended: 1.157, Last appended by leader: 157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: true } }
I20240503 13:59:16.042197 5956 ts_tablet_manager.cc:1398] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:16.042830 3890 tablet_copy_service.cc:342] P 0c6ca23f9d9b4c91a6347c0924674bdd: Request end of tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-e572db34c8844384b728068168645ecc received from {username='slave'} at 127.0.99.133:48411
I20240503 13:59:16.042944 3890 tablet_copy_service.cc:433] P 0c6ca23f9d9b4c91a6347c0924674bdd: ending tablet copy session 14a79ecc8c2c45adac20a5239cd45fd2-e572db34c8844384b728068168645ecc on tablet e572db34c8844384b728068168645ecc with peer 14a79ecc8c2c45adac20a5239cd45fd2
I20240503 13:59:16.260032 4974 raft_consensus.cc:1203] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.122->[1.123-1.143] Dedup: 1.137->[1.138-1.143]
I20240503 13:59:16.262234 5240 raft_consensus.cc:1203] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.126->[1.127-1.169] Dedup: 1.146->[1.147-1.169]
I20240503 13:59:16.303911 5936 raft_consensus.cc:1050] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 to VOTER
I20240503 13:59:16.304307 5936 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 1.145, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:16.305927 4974 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240503 13:59:16.306344 5367 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 144, Time since last communication: 0.000s
I20240503 13:59:16.307535 4843 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240503 13:59:16.308385 5490 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 144, Time since last communication: 0.000s
I20240503 13:59:16.308743 5490 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.146: config changed from index 123 to 146, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.309803 4973 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 123 to 146, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.312865 3458 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 reported cstate change: config changed from index 123 to 146, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.315660 4843 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 123 to 146, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.326944 4756 raft_consensus.cc:1050] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 07bdad76c7bd40afb3147833b3305924 to VOTER
I20240503 13:59:16.327327 4756 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:16.328717 5106 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 13:59:16.329716 5240 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 13:59:16.330952 5936 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 13:59:16.331532 5942 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 13:59:16.333051 4756 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 127 to 171, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:16.333503 5240 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 127 to 171, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:16.336426 3459 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: config changed from index 127 to 171, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:16.340046 5106 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 127 to 171, 07bdad76c7bd40afb3147833b3305924 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:16.389865 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd23523f46044bb7b7e4f44056687c43"
dest_uuid: "3812cd29374a4b3485883810d0774867"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49830
I20240503 13:59:16.390116 3737 raft_consensus.cc:595] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:16.393519 5942 raft_consensus.cc:979] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: : Instructing follower 4e553631c0564b078cefd1cd1cdcc90d to start an election
I20240503 13:59:16.393625 5942 raft_consensus.cc:1067] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Signalling peer 4e553631c0564b078cefd1cd1cdcc90d to start an election
I20240503 13:59:16.394263 4843 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd23523f46044bb7b7e4f44056687c43"
dest_uuid: "4e553631c0564b078cefd1cd1cdcc90d"
from {username='slave'} at 127.0.99.130:49535
I20240503 13:59:16.394481 4843 raft_consensus.cc:482] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:16.394615 4843 raft_consensus.cc:3037] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.395864 4843 raft_consensus.cc:504] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:16.397164 4843 leader_election.cc:288] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [CANDIDATE]: Term 2 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107), 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785)
I20240503 13:59:16.399190 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd23523f46044bb7b7e4f44056687c43" candidate_uuid: "4e553631c0564b078cefd1cd1cdcc90d" candidate_term: 2 candidate_status { last_received { term: 1 index: 149 } } ignore_live_leader: true dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:16.399333 3737 raft_consensus.cc:3032] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:16.399375 3737 raft_consensus.cc:729] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:16.399518 3737 consensus_queue.cc:260] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:16.399700 3737 raft_consensus.cc:3037] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.400708 3737 raft_consensus.cc:2454] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e553631c0564b078cefd1cd1cdcc90d in term 2.
I20240503 13:59:16.401346 4777 leader_election.cc:302] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3812cd29374a4b3485883810d0774867, 4e553631c0564b078cefd1cd1cdcc90d; no voters:
I20240503 13:59:16.401616 6034 raft_consensus.cc:2781] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:16.401788 6034 raft_consensus.cc:686] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 2 LEADER]: Becoming Leader. State: Replica: 4e553631c0564b078cefd1cd1cdcc90d, State: Running, Role: LEADER
I20240503 13:59:16.402134 6034 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:16.405545 3463 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d reported cstate change: term changed from 1 to 2, leader changed from 3812cd29374a4b3485883810d0774867 (127.0.99.130) to 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132). New cstate: current_term: 2 leader_uuid: "4e553631c0564b078cefd1cd1cdcc90d" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:16.437289 4975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd23523f46044bb7b7e4f44056687c43" candidate_uuid: "4e553631c0564b078cefd1cd1cdcc90d" candidate_term: 2 candidate_status { last_received { term: 1 index: 149 } } ignore_live_leader: true dest_uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
I20240503 13:59:16.437565 4975 raft_consensus.cc:3037] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.438933 4975 raft_consensus.cc:2454] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e553631c0564b078cefd1cd1cdcc90d in term 2.
W20240503 13:59:16.477892 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49540: Illegal state: replica 3812cd29374a4b3485883810d0774867 is not leader of this config: current role FOLLOWER
I20240503 13:59:16.509464 3737 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Refusing update from remote peer 4e553631c0564b078cefd1cd1cdcc90d: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 2 index: 151. (index mismatch)
I20240503 13:59:16.510582 4974 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Refusing update from remote peer 4e553631c0564b078cefd1cd1cdcc90d: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 2 index: 151. (index mismatch)
I20240503 13:59:16.511335 6034 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 13:59:16.511765 6034 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20240503 13:59:16.520601 5914 mvcc.cc:205] Tried to move back new op lower bound from 7023594522657693696 to 7023594522226061312. Current Snapshot: MvccSnapshot[applied={T|T < 7023594522657693696 or (T in {7023594522657693696})}]
I20240503 13:59:16.532727 4843 consensus_queue.cc:237] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 151, Committed index: 151, Last appended: 2.151, Last appended by leader: 149, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:16.541538 4973 raft_consensus.cc:1261] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Refusing update from remote peer 4e553631c0564b078cefd1cd1cdcc90d: Log matching property violated. Preceding OpId in replica: term: 2 index: 151. Preceding OpId from leader: term: 2 index: 152. (index mismatch)
I20240503 13:59:16.542450 6050 consensus_queue.cc:1035] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 152, Last known committed idx: 151, Time since last communication: 0.000s
I20240503 13:59:16.544703 6046 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d [term 2 LEADER]: Committing config change with OpId 2.152: config changed from index 146 to 152, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.571059 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd23523f46044bb7b7e4f44056687c43 with cas_config_opid_index 146: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:16.572934 3463 catalog_manager.cc:5484] T bd23523f46044bb7b7e4f44056687c43 P 4e553631c0564b078cefd1cd1cdcc90d reported cstate change: config changed from index 146 to 152, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "4e553631c0564b078cefd1cd1cdcc90d" committed_config { opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:16.573697 4973 raft_consensus.cc:2932] T bd23523f46044bb7b7e4f44056687c43 P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Committing config change with OpId 2.152: config changed from index 146 to 152, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.581964 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet bd23523f46044bb7b7e4f44056687c43 with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 152) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:16.587478 4974 raft_consensus.cc:1203] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.149->[1.150-1.190] Dedup: 1.157->[1.158-1.190]
I20240503 13:59:16.594815 6052 tablet_replica.cc:330] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:16.595806 6052 raft_consensus.cc:2229] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:16.596045 6052 raft_consensus.cc:2258] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:16.596907 6052 ts_tablet_manager.cc:1875] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:16.610118 6052 ts_tablet_manager.cc:1888] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.151
I20240503 13:59:16.614207 4622 raft_consensus.cc:1050] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: attempting to promote NON_VOTER 14a79ecc8c2c45adac20a5239cd45fd2 to VOTER
I20240503 13:59:16.614624 4622 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 190, Committed index: 190, Last appended: 1.191, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:16.615767 6052 log.cc:1196] T bd23523f46044bb7b7e4f44056687c43 P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:16.616644 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet bd23523f46044bb7b7e4f44056687c43 (table rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]) successfully deleted
I20240503 13:59:16.617424 4973 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 LEARNER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 1 index: 192. (index mismatch)
I20240503 13:59:16.618891 5377 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 190, Time since last communication: 0.000s
I20240503 13:59:16.621775 5377 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Committing config change with OpId 1.192: config changed from index 150 to 192, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.622232 4974 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index 150 to 192, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.624943 3457 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd reported cstate change: config changed from index 150 to 192, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" committed_config { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:16.634531 4842 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Refusing update from remote peer 0c6ca23f9d9b4c91a6347c0924674bdd: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 1 index: 192. (index mismatch)
I20240503 13:59:16.636515 5348 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 190, Time since last communication: 0.000s
I20240503 13:59:16.644192 4842 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index 150 to 192, 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:16.706313 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49afc417528049839897ee5600b954e8"
dest_uuid: "3812cd29374a4b3485883810d0774867"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49840
I20240503 13:59:16.706514 3737 raft_consensus.cc:595] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:16.719992 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.728811 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.735647 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
I20240503 13:59:16.739816 3869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e572db34c8844384b728068168645ecc"
dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57396
I20240503 13:59:16.739987 3869 raft_consensus.cc:595] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:16.744817 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.752146 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.759364 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.770057 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.770715 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.776547 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.780644 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.781430 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.789520 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.791174 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
I20240503 13:59:16.794099 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49afc417528049839897ee5600b954e8"
dest_uuid: "3812cd29374a4b3485883810d0774867"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49860
I20240503 13:59:16.794272 3737 raft_consensus.cc:595] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:16.800417 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.806295 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.808353 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
I20240503 13:59:16.819351 3869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e572db34c8844384b728068168645ecc"
dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57414
I20240503 13:59:16.819536 3869 raft_consensus.cc:595] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:16.820243 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.822553 3697 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.830078 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.839082 3697 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.840765 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.856770 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.856907 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.868716 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.875707 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.882055 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.894435 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.899556 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.913810 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.915360 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.932637 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.935243 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.951380 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.954599 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.974113 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.976279 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:16.994406 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:16.999051 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:17.018976 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:17.024180 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:17.044943 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:17.046365 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
I20240503 13:59:17.057380 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49afc417528049839897ee5600b954e8"
dest_uuid: "3812cd29374a4b3485883810d0774867"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49878
I20240503 13:59:17.057528 3737 raft_consensus.cc:595] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:17.068389 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:17.076203 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
W20240503 13:59:17.097767 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
W20240503 13:59:17.105669 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
I20240503 13:59:17.116405 3869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e572db34c8844384b728068168645ecc"
dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57432
I20240503 13:59:17.116557 3869 raft_consensus.cc:595] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:17.123024 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Service unavailable: leader transfer in progress
I20240503 13:59:17.125437 5348 raft_consensus.cc:979] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: : Instructing follower 14a79ecc8c2c45adac20a5239cd45fd2 to start an election
I20240503 13:59:17.125535 5348 raft_consensus.cc:1067] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Signalling peer 14a79ecc8c2c45adac20a5239cd45fd2 to start an election
I20240503 13:59:17.126924 4972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e572db34c8844384b728068168645ecc"
dest_uuid: "14a79ecc8c2c45adac20a5239cd45fd2"
from {username='slave'} at 127.0.99.131:56439
I20240503 13:59:17.127118 4972 raft_consensus.cc:482] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:17.127184 4972 raft_consensus.cc:3037] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:17.128300 4972 raft_consensus.cc:504] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
W20240503 13:59:17.131501 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
I20240503 13:59:17.138239 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572db34c8844384b728068168645ecc" candidate_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 195 } } ignore_live_leader: true dest_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd"
I20240503 13:59:17.138414 3869 raft_consensus.cc:3032] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:17.138465 3869 raft_consensus.cc:729] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c6ca23f9d9b4c91a6347c0924674bdd, State: Running, Role: LEADER
I20240503 13:59:17.138598 3869 consensus_queue.cc:260] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 1.195, Last appended by leader: 195, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:17.138756 3869 raft_consensus.cc:3037] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:17.138859 4972 leader_election.cc:288] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [CANDIDATE]: Term 2 election: Requested vote from peers 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619), 4e553631c0564b078cefd1cd1cdcc90d (127.0.99.132:36697)
I20240503 13:59:17.139936 3869 raft_consensus.cc:2454] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a79ecc8c2c45adac20a5239cd45fd2 in term 2.
I20240503 13:59:17.140458 4910 leader_election.cc:302] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6ca23f9d9b4c91a6347c0924674bdd, 14a79ecc8c2c45adac20a5239cd45fd2; no voters:
W20240503 13:59:17.148741 3828 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57118: Illegal state: replica 0c6ca23f9d9b4c91a6347c0924674bdd is not leader of this config: current role FOLLOWER
I20240503 13:59:17.155078 6125 raft_consensus.cc:2781] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:17.155318 6125 raft_consensus.cc:686] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 LEADER]: Becoming Leader. State: Replica: 14a79ecc8c2c45adac20a5239cd45fd2, State: Running, Role: LEADER
I20240503 13:59:17.155846 6125 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 1.195, Last appended by leader: 195, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:17.155920 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e572db34c8844384b728068168645ecc" candidate_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 195 } } ignore_live_leader: true dest_uuid: "4e553631c0564b078cefd1cd1cdcc90d"
I20240503 13:59:17.156098 4842 raft_consensus.cc:3037] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:17.159396 3463 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 reported cstate change: term changed from 1 to 2, leader changed from 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) to 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" committed_config { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:17.161343 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Service unavailable: leader transfer in progress
I20240503 13:59:17.162712 4842 raft_consensus.cc:2454] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a79ecc8c2c45adac20a5239cd45fd2 in term 2.
I20240503 13:59:17.187412 5936 raft_consensus.cc:979] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: : Instructing follower 07bdad76c7bd40afb3147833b3305924 to start an election
I20240503 13:59:17.187574 5936 raft_consensus.cc:1067] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Signalling peer 07bdad76c7bd40afb3147833b3305924 to start an election
I20240503 13:59:17.188041 5243 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49afc417528049839897ee5600b954e8"
dest_uuid: "07bdad76c7bd40afb3147833b3305924"
from {username='slave'} at 127.0.99.130:60327
I20240503 13:59:17.188261 5243 raft_consensus.cc:482] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:17.188335 5243 raft_consensus.cc:3037] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:17.189549 5243 raft_consensus.cc:504] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:17.190826 5243 leader_election.cc:288] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [CANDIDATE]: Term 2 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107), 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797)
I20240503 13:59:17.191939 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49afc417528049839897ee5600b954e8" candidate_uuid: "07bdad76c7bd40afb3147833b3305924" candidate_term: 2 candidate_status { last_received { term: 1 index: 186 } } ignore_live_leader: true dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:17.192078 3737 raft_consensus.cc:3032] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:17.192123 3737 raft_consensus.cc:729] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:17.192265 3737 consensus_queue.cc:260] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 1.186, Last appended by leader: 186, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:17.192440 3737 raft_consensus.cc:3037] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:17.193354 3698 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49590: Illegal state: replica 3812cd29374a4b3485883810d0774867 is not leader of this config: current role FOLLOWER
I20240503 13:59:17.193389 3737 raft_consensus.cc:2454] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bdad76c7bd40afb3147833b3305924 in term 2.
I20240503 13:59:17.193953 5173 leader_election.cc:302] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bdad76c7bd40afb3147833b3305924, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:17.194442 6132 raft_consensus.cc:2781] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:17.194666 6132 raft_consensus.cc:686] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Becoming Leader. State: Replica: 07bdad76c7bd40afb3147833b3305924, State: Running, Role: LEADER
I20240503 13:59:17.195065 6132 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 1.186, Last appended by leader: 186, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:17.199690 3463 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: term changed from 1 to 2, leader changed from 3812cd29374a4b3485883810d0774867 (127.0.99.130) to 07bdad76c7bd40afb3147833b3305924 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:17.216567 5106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49afc417528049839897ee5600b954e8" candidate_uuid: "07bdad76c7bd40afb3147833b3305924" candidate_term: 2 candidate_status { last_received { term: 1 index: 186 } } ignore_live_leader: true dest_uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
I20240503 13:59:17.216804 5106 raft_consensus.cc:3037] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:17.218006 5106 raft_consensus.cc:2454] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bdad76c7bd40afb3147833b3305924 in term 2.
I20240503 13:59:17.229771 3869 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Refusing update from remote peer 14a79ecc8c2c45adac20a5239cd45fd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 195. Preceding OpId from leader: term: 2 index: 197. (index mismatch)
I20240503 13:59:17.229941 4842 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 2 FOLLOWER]: Refusing update from remote peer 14a79ecc8c2c45adac20a5239cd45fd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 195. Preceding OpId from leader: term: 2 index: 197. (index mismatch)
I20240503 13:59:17.230671 6126 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I20240503 13:59:17.231098 6125 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6ca23f9d9b4c91a6347c0924674bdd" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41619 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I20240503 13:59:17.237066 5586 mvcc.cc:205] Tried to move back new op lower bound from 7023594525608415232 to 7023594525313822720. Current Snapshot: MvccSnapshot[applied={T|T < 7023594525608415232 or (T in {7023594525608415232})}]
I20240503 13:59:17.255885 4972 consensus_queue.cc:237] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 2.198, Last appended by leader: 195, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } }
I20240503 13:59:17.269140 4842 raft_consensus.cc:1261] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 2 FOLLOWER]: Refusing update from remote peer 14a79ecc8c2c45adac20a5239cd45fd2: Log matching property violated. Preceding OpId in replica: term: 2 index: 198. Preceding OpId from leader: term: 2 index: 199. (index mismatch)
I20240503 13:59:17.269605 6139 consensus_queue.cc:1035] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240503 13:59:17.275442 6126 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 [term 2 LEADER]: Committing config change with OpId 2.199: config changed from index 192 to 199, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:17.276149 4842 raft_consensus.cc:2932] T e572db34c8844384b728068168645ecc P 4e553631c0564b078cefd1cd1cdcc90d [term 2 FOLLOWER]: Committing config change with OpId 2.199: config changed from index 192 to 199, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } } }
I20240503 13:59:17.278312 3453 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e572db34c8844384b728068168645ecc with cas_config_opid_index 192: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:17.278805 3457 catalog_manager.cc:5484] T e572db34c8844384b728068168645ecc P 14a79ecc8c2c45adac20a5239cd45fd2 reported cstate change: config changed from index 192 to 199, VOTER 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "4e553631c0564b078cefd1cd1cdcc90d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:17.284693 5106 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 1 index: 186. Preceding OpId from leader: term: 2 index: 188. (index mismatch)
I20240503 13:59:17.285025 3737 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 1 index: 186. Preceding OpId from leader: term: 2 index: 188. (index mismatch)
I20240503 13:59:17.285180 3849 tablet_service.cc:1514] Processing DeleteTablet for tablet e572db34c8844384b728068168645ecc with delete_type TABLET_DATA_TOMBSTONED (TS 0c6ca23f9d9b4c91a6347c0924674bdd not found in new config with opid_index 199) from {username='slave'} at 127.0.0.1:46164
I20240503 13:59:17.285648 6133 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I20240503 13:59:17.286074 6132 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I20240503 13:59:17.286692 6143 tablet_replica.cc:330] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: stopping tablet replica
I20240503 13:59:17.287119 6143 raft_consensus.cc:2229] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:17.290336 5858 mvcc.cc:205] Tried to move back new op lower bound from 7023594525831979008 to 7023594525481398272. Current Snapshot: MvccSnapshot[applied={T|T < 7023594525831979008}]
I20240503 13:59:17.292856 6143 raft_consensus.cc:2258] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:17.295325 6143 ts_tablet_manager.cc:1875] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:17.310067 6143 ts_tablet_manager.cc:1888] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.198
I20240503 13:59:17.317466 6143 log.cc:1196] T e572db34c8844384b728068168645ecc P 0c6ca23f9d9b4c91a6347c0924674bdd: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e572db34c8844384b728068168645ecc
I20240503 13:59:17.318168 3452 catalog_manager.cc:4830] TS 0c6ca23f9d9b4c91a6347c0924674bdd (127.0.99.131:41619): tablet e572db34c8844384b728068168645ecc (table rebalance_test_table_2 [id=25d077c1a2254f8fa8f4273dfb74ed83]) successfully deleted
I20240503 13:59:17.319882 5243 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 2.188, Last appended by leader: 186, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } }
I20240503 13:59:17.322474 5105 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 2 index: 188. Preceding OpId from leader: term: 2 index: 189. (index mismatch)
I20240503 13:59:17.328887 6144 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240503 13:59:17.338053 6132 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Committing config change with OpId 2.189: config changed from index 171 to 189, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:17.340325 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49afc417528049839897ee5600b954e8 with cas_config_opid_index 171: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:17.341886 3463 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: config changed from index 171 to 189, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:17.345839 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet 49afc417528049839897ee5600b954e8 with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 189) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:17.351929 6150 tablet_replica.cc:330] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:17.352099 6150 raft_consensus.cc:2229] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:17.353801 5106 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Committing config change with OpId 2.189: config changed from index 171 to 189, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } }
I20240503 13:59:17.354123 6150 raft_consensus.cc:2258] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:17.356006 6150 ts_tablet_manager.cc:1875] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:17.359959 6150 ts_tablet_manager.cc:1888] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.188
I20240503 13:59:17.360127 6150 log.cc:1196] T 49afc417528049839897ee5600b954e8 P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/49afc417528049839897ee5600b954e8
I20240503 13:59:17.361582 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet 49afc417528049839897ee5600b954e8 (table rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]) successfully deleted
I20240503 13:59:17.707151 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:17.715786 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:17.741099 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:17.756979 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:17.759171 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:17.778908 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:17.785514 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.124012 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:18.129050 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.158092 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:18.158366 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:18.167275 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.181640 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.288796 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.900692 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.910037 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:18.929224 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.930670 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:18.943790 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:18.948433 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:18.965235 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.327100 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.329722 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:19.343017 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.352228 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:19.390178 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:19.397732 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.408782 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.584085 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.585196 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:19.586737 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.595463 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:19.596344 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:19.615615 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:19.617256 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
5956e2ef385b4f099bcf90b9d7156148 | 127.0.99.190:43017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:42243 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
07bdad76c7bd40afb3147833b3305924 | 127.0.99.135:43643 | HEALTHY | <none> | 1 | 0
0c6ca23f9d9b4c91a6347c0924674bdd | 127.0.99.131:41619 | HEALTHY | <none> | 4 | 0
14a79ecc8c2c45adac20a5239cd45fd2 | 127.0.99.133:39785 | HEALTHY | <none> | 1 | 0
3812cd29374a4b3485883810d0774867 | 127.0.99.130:42107 | HEALTHY | <none> | 5 | 0
4e553631c0564b078cefd1cd1cdcc90d | 127.0.99.132:36697 | HEALTHY | <none> | 1 | 0
6d1a97f876fc4f8698f737fcf1b782ad | 127.0.99.134:43797 | HEALTHY | <none> | 1 | 0
bc41cca530f54259aca15e34563208dc | 127.0.99.129:43281 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43281
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:42107
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:41619
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:36697
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:39785
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:43797
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:43643
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43281
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:42107
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:41619
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:36697
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:39785
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:43797
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:43643
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:42243 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:59:19.642781 398 log_verifier.cc:126] Checking tablet 19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:19.698767 398 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:19.698900 398 log_verifier.cc:126] Checking tablet 2e7fa1300123438896b7be116d5560fa
I20240503 13:59:19.745831 398 log_verifier.cc:177] Verified matching terms for 220 ops in tablet 2e7fa1300123438896b7be116d5560fa
I20240503 13:59:19.745946 398 log_verifier.cc:126] Checking tablet 37a9c633962b40f1b42376cf597ad415
I20240503 13:59:19.795123 398 log_verifier.cc:177] Verified matching terms for 252 ops in tablet 37a9c633962b40f1b42376cf597ad415
I20240503 13:59:19.795239 398 log_verifier.cc:126] Checking tablet 480d84e5600d40e7b8c7fe001ba33c9f
I20240503 13:59:19.846473 398 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 480d84e5600d40e7b8c7fe001ba33c9f
I20240503 13:59:19.846603 398 log_verifier.cc:126] Checking tablet 49afc417528049839897ee5600b954e8
I20240503 13:59:19.897078 398 log_verifier.cc:177] Verified matching terms for 257 ops in tablet 49afc417528049839897ee5600b954e8
I20240503 13:59:19.897212 398 log_verifier.cc:126] Checking tablet 62fbc3876d064aa0a8e7f8f71fb3f35f
I20240503 13:59:19.949927 398 log_verifier.cc:177] Verified matching terms for 272 ops in tablet 62fbc3876d064aa0a8e7f8f71fb3f35f
I20240503 13:59:19.950086 398 log_verifier.cc:126] Checking tablet 809441335aa64f959ef084cb3c8b2434
I20240503 13:59:20.011672 398 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 809441335aa64f959ef084cb3c8b2434
I20240503 13:59:20.011860 398 log_verifier.cc:126] Checking tablet 8df17b9134914ca3a2c04800435b3697
I20240503 13:59:20.028617 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 8df17b9134914ca3a2c04800435b3697
I20240503 13:59:20.028702 398 log_verifier.cc:126] Checking tablet 97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:20.081215 398 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:20.081369 398 log_verifier.cc:126] Checking tablet adfa836cb94c4050bdb65e007b232f2b
I20240503 13:59:20.135821 398 log_verifier.cc:177] Verified matching terms for 284 ops in tablet adfa836cb94c4050bdb65e007b232f2b
I20240503 13:59:20.135969 398 log_verifier.cc:126] Checking tablet bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:20.183245 398 log_verifier.cc:177] Verified matching terms for 236 ops in tablet bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:20.183374 398 log_verifier.cc:126] Checking tablet c1c58189e2df41c3b8b4a0924ae42a03
I20240503 13:59:20.236734 398 log_verifier.cc:177] Verified matching terms for 288 ops in tablet c1c58189e2df41c3b8b4a0924ae42a03
I20240503 13:59:20.236877 398 log_verifier.cc:126] Checking tablet dcf42adb4ddc4a25907a5c5fa94e31b7
I20240503 13:59:20.292912 398 log_verifier.cc:177] Verified matching terms for 303 ops in tablet dcf42adb4ddc4a25907a5c5fa94e31b7
I20240503 13:59:20.293057 398 log_verifier.cc:126] Checking tablet e572db34c8844384b728068168645ecc
I20240503 13:59:20.344034 398 log_verifier.cc:177] Verified matching terms for 260 ops in tablet e572db34c8844384b728068168645ecc
I20240503 13:59:20.344169 398 log_verifier.cc:126] Checking tablet e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:20.405875 398 log_verifier.cc:177] Verified matching terms for 291 ops in tablet e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:20.406040 398 log_verifier.cc:126] Checking tablet fc8c685a1dd4496b9be2b1e09d237371
I20240503 13:59:20.461431 398 log_verifier.cc:177] Verified matching terms for 293 ops in tablet fc8c685a1dd4496b9be2b1e09d237371
I20240503 13:59:20.463529 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--builtin_ntp_servers=127.0.99.148:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:20.580538 6366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:20.580775 6366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:20.580818 6366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:20.584187 6366 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:20.584270 6366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:20.584342 6366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:59:20.588297 6366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43017
--never_fsync=true
--heap_profile_path=/tmp/kudu.6366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:20.589267 6366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:20.590497 6366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:20.599975 6375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:20.600764 6372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:20.601325 6373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:20.602155 6366 server_base.cc:1034] running on GCE node
I20240503 13:59:20.602556 6366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:20.603202 6366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:20.604360 6366 hybrid_clock.cc:648] HybridClock initialized: now 1714744760604337 us; error 42 us; skew 500 ppm
I20240503 13:59:20.606390 6366 webserver.cc:460] Webserver started at http://127.0.99.136:34153/ using document root <none> and password file <none>
I20240503 13:59:20.606963 6366 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:20.607015 6366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:20.607187 6366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:20.608981 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0b76761b17fc43d1b1a57866605abd56"
format_stamp: "Formatted at 2024-05-03 13:59:20 on dist-test-slave-6fwv"
I20240503 13:59:20.609467 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0b76761b17fc43d1b1a57866605abd56"
format_stamp: "Formatted at 2024-05-03 13:59:20 on dist-test-slave-6fwv"
I20240503 13:59:20.613092 6366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240503 13:59:20.615783 6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:20.617002 6366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:20.617142 6366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0b76761b17fc43d1b1a57866605abd56"
format_stamp: "Formatted at 2024-05-03 13:59:20 on dist-test-slave-6fwv"
I20240503 13:59:20.617261 6366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:20.635485 6366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:20.636294 6366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:20.636476 6366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:20.637164 6366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:20.638182 6366 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:20.638235 6366 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:20.638288 6366 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:20.638311 6366 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:20.647195 6366 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:36697
I20240503 13:59:20.647219 6494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:36697 every 8 connection(s)
I20240503 13:59:20.648283 6366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:59:20.650277 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 6366
I20240503 13:59:20.650396 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:59:20.658684 6495 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43017
I20240503 13:59:20.658965 6495 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:20.659613 6495 heartbeater.cc:511] Master 127.0.99.190:43017 requested a full tablet report, sending...
I20240503 13:59:20.660768 3457 ts_manager.cc:194] Registered new tserver with Master: 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136:36697)
I20240503 13:59:20.661368 3457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:47701
W20240503 13:59:20.850873 4889 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.99.132:36697: Invalid argument: socket address is ambiguous
I20240503 13:59:20.858147 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:20.858544 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:20.861627 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:20.867285 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:20.869001 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:20.877274 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:20.889333 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:20.901007 6429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:20.991963 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:20.993450 5217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:20.999363 3717 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 13:59:21.002998 4885 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.99.132:36697
I20240503 13:59:21.003749 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.009322 3587 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:21.009306 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.010139 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.022717 6429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:21.063905 3737 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.065775 4974 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240503 13:59:21.066197 6591 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240503 13:59:21.067965 6224 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 79 to 303, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
I20240503 13:59:21.070067 4974 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 79 to 303, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
W20240503 13:59:21.071028 3672 consensus_peers.cc:485] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 -> Peer 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136:36697): Couldn't send request to peer 0b76761b17fc43d1b1a57866605abd56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19ebb671e56f4e29adb6a2c0a7c6b00e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:21.071818 3462 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 79 to 303, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.079708 3603 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.081521 5240 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20240503 13:59:21.081951 6604 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20240503 13:59:21.083904 5879 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 111 to 271, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
W20240503 13:59:21.085391 3541 consensus_peers.cc:485] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc -> Peer 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136:36697): Couldn't send request to peer 0b76761b17fc43d1b1a57866605abd56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97d59e7bd0da44d9a7af791f2477b183. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:21.085652 5240 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 111 to 271, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
I20240503 13:59:21.086382 3463 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 111 to 271, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.095368 3737 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.096879 5106 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
W20240503 13:59:21.096951 3672 consensus_peers.cc:485] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 -> Peer 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136:36697): Couldn't send request to peer 0b76761b17fc43d1b1a57866605abd56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e967e4257bda4e4fa9b3fdfadd22368f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:21.097285 6224 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20240503 13:59:21.099027 6591 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.292: config changed from index 110 to 292, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
I20240503 13:59:21.099536 5106 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 110 to 292, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
I20240503 13:59:21.101816 3463 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 110 to 292, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.112280 5240 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 186, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.114042 5106 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20240503 13:59:21.114545 6621 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 13:59:21.116585 6500 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 189 to 258, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
I20240503 13:59:21.117316 5106 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 189 to 258, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } } }
W20240503 13:59:21.118901 5173 consensus_peers.cc:485] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 -> Peer 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136:36697): Couldn't send request to peer 0b76761b17fc43d1b1a57866605abd56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49afc417528049839897ee5600b954e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:21.119374 3462 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: config changed from index 189 to 258, NON_VOTER 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:21.553892 6664 ts_tablet_manager.cc:902] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Initiating tablet copy from peer 07bdad76c7bd40afb3147833b3305924 (127.0.99.135:43643)
I20240503 13:59:21.555385 6664 tablet_copy_client.cc:308] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:43643
I20240503 13:59:21.559357 5266 tablet_copy_service.cc:140] P 07bdad76c7bd40afb3147833b3305924: Received BeginTabletCopySession request for tablet 49afc417528049839897ee5600b954e8 from peer 0b76761b17fc43d1b1a57866605abd56 ({username='slave'} at 127.0.99.136:58773)
I20240503 13:59:21.559561 5266 tablet_copy_service.cc:161] P 07bdad76c7bd40afb3147833b3305924: Beginning new tablet copy session on tablet 49afc417528049839897ee5600b954e8 from peer 0b76761b17fc43d1b1a57866605abd56 at {username='slave'} at 127.0.99.136:58773: session id = 0b76761b17fc43d1b1a57866605abd56-49afc417528049839897ee5600b954e8
I20240503 13:59:21.560953 5266 tablet_copy_source_session.cc:205] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:21.564658 6664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49afc417528049839897ee5600b954e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.570096 6664 tablet_copy_client.cc:783] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:21.570426 6664 tablet_copy_client.cc:647] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:21.572718 6664 tablet_copy_client.cc:523] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:21.574771 6664 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Bootstrap starting.
I20240503 13:59:21.597216 6664 log.cc:824] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:21.606209 6691 ts_tablet_manager.cc:902] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:21.606817 6691 tablet_copy_client.cc:308] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:21.610802 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e from peer 0b76761b17fc43d1b1a57866605abd56 ({username='slave'} at 127.0.99.136:55675)
I20240503 13:59:21.610920 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet 19ebb671e56f4e29adb6a2c0a7c6b00e from peer 0b76761b17fc43d1b1a57866605abd56 at {username='slave'} at 127.0.99.136:55675: session id = 0b76761b17fc43d1b1a57866605abd56-19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:21.611649 3759 tablet_copy_source_session.cc:205] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:21.612629 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.615056 6691 tablet_copy_client.cc:783] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:21.615213 6691 tablet_copy_client.cc:647] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:21.617928 6691 tablet_copy_client.cc:523] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:21.619295 6691 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: Bootstrap starting.
W20240503 13:59:21.656196 6490 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.99.136:36697
I20240503 13:59:21.663744 6495 heartbeater.cc:503] Master 127.0.99.190:43017 was elected leader, sending a full tablet report...
I20240503 13:59:21.688195 6697 ts_tablet_manager.cc:902] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: Initiating tablet copy from peer bc41cca530f54259aca15e34563208dc (127.0.99.129:43281)
I20240503 13:59:21.688820 6697 tablet_copy_client.cc:308] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43281
I20240503 13:59:21.693229 3628 tablet_copy_service.cc:140] P bc41cca530f54259aca15e34563208dc: Received BeginTabletCopySession request for tablet 97d59e7bd0da44d9a7af791f2477b183 from peer 0b76761b17fc43d1b1a57866605abd56 ({username='slave'} at 127.0.99.136:44561)
I20240503 13:59:21.693408 3628 tablet_copy_service.cc:161] P bc41cca530f54259aca15e34563208dc: Beginning new tablet copy session on tablet 97d59e7bd0da44d9a7af791f2477b183 from peer 0b76761b17fc43d1b1a57866605abd56 at {username='slave'} at 127.0.99.136:44561: session id = 0b76761b17fc43d1b1a57866605abd56-97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:21.694613 3628 tablet_copy_source_session.cc:205] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:21.695696 6697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d59e7bd0da44d9a7af791f2477b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.698177 6697 tablet_copy_client.cc:783] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:21.698503 6697 tablet_copy_client.cc:647] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:21.700409 6697 tablet_copy_client.cc:523] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:21.701932 6697 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: Bootstrap starting.
I20240503 13:59:21.712888 6702 ts_tablet_manager.cc:902] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: Initiating tablet copy from peer 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107)
I20240503 13:59:21.713584 6702 tablet_copy_client.cc:308] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42107
I20240503 13:59:21.714259 3759 tablet_copy_service.cc:140] P 3812cd29374a4b3485883810d0774867: Received BeginTabletCopySession request for tablet e967e4257bda4e4fa9b3fdfadd22368f from peer 0b76761b17fc43d1b1a57866605abd56 ({username='slave'} at 127.0.99.136:55675)
I20240503 13:59:21.714383 3759 tablet_copy_service.cc:161] P 3812cd29374a4b3485883810d0774867: Beginning new tablet copy session on tablet e967e4257bda4e4fa9b3fdfadd22368f from peer 0b76761b17fc43d1b1a57866605abd56 at {username='slave'} at 127.0.99.136:55675: session id = 0b76761b17fc43d1b1a57866605abd56-e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:21.715463 3759 tablet_copy_source_session.cc:205] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:21.716686 6702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e967e4257bda4e4fa9b3fdfadd22368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:21.719637 6702 tablet_copy_client.cc:783] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:21.719902 6702 tablet_copy_client.cc:647] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:21.722644 6702 tablet_copy_client.cc:523] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:21.724176 6702 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: Bootstrap starting.
I20240503 13:59:21.764958 6664 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Bootstrap replayed 1/1 log segments. Stats: ops{read=258 overwritten=0 applied=258 ignored=0} inserts{seen=882 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:21.765408 6664 tablet_bootstrap.cc:492] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Bootstrap complete.
I20240503 13:59:21.765810 6664 ts_tablet_manager.cc:1367] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Time spent bootstrapping tablet: real 0.191s user 0.163s sys 0.010s
I20240503 13:59:21.768667 6664 raft_consensus.cc:348] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.768958 6664 raft_consensus.cc:729] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0b76761b17fc43d1b1a57866605abd56, State: Initialized, Role: LEARNER
I20240503 13:59:21.769364 6664 consensus_queue.cc:260] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 258, Last appended: 2.258, Last appended by leader: 258, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.770388 6664 ts_tablet_manager.cc:1398] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240503 13:59:21.773131 5266 tablet_copy_service.cc:342] P 07bdad76c7bd40afb3147833b3305924: Request end of tablet copy session 0b76761b17fc43d1b1a57866605abd56-49afc417528049839897ee5600b954e8 received from {username='slave'} at 127.0.99.136:58773
I20240503 13:59:21.773250 5266 tablet_copy_service.cc:433] P 07bdad76c7bd40afb3147833b3305924: ending tablet copy session 0b76761b17fc43d1b1a57866605abd56-49afc417528049839897ee5600b954e8 on tablet 49afc417528049839897ee5600b954e8 with peer 0b76761b17fc43d1b1a57866605abd56
I20240503 13:59:21.832801 6691 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: Bootstrap replayed 1/1 log segments. Stats: ops{read=303 overwritten=0 applied=303 ignored=0} inserts{seen=931 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:21.833143 6691 tablet_bootstrap.cc:492] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: Bootstrap complete.
I20240503 13:59:21.833386 6691 ts_tablet_manager.cc:1367] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: Time spent bootstrapping tablet: real 0.214s user 0.202s sys 0.004s
I20240503 13:59:21.834201 6691 raft_consensus.cc:348] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.834347 6691 raft_consensus.cc:729] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b76761b17fc43d1b1a57866605abd56, State: Initialized, Role: LEARNER
I20240503 13:59:21.834466 6691 consensus_queue.cc:260] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 303, Last appended: 1.303, Last appended by leader: 303, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.835354 6691 ts_tablet_manager.cc:1398] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 13:59:21.835899 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 0b76761b17fc43d1b1a57866605abd56-19ebb671e56f4e29adb6a2c0a7c6b00e received from {username='slave'} at 127.0.99.136:55675
I20240503 13:59:21.835980 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 0b76761b17fc43d1b1a57866605abd56-19ebb671e56f4e29adb6a2c0a7c6b00e on tablet 19ebb671e56f4e29adb6a2c0a7c6b00e with peer 0b76761b17fc43d1b1a57866605abd56
I20240503 13:59:21.906355 6697 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: Bootstrap replayed 1/1 log segments. Stats: ops{read=271 overwritten=0 applied=271 ignored=0} inserts{seen=899 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:21.906654 6697 tablet_bootstrap.cc:492] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: Bootstrap complete.
I20240503 13:59:21.906849 6697 ts_tablet_manager.cc:1367] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: Time spent bootstrapping tablet: real 0.205s user 0.189s sys 0.008s
I20240503 13:59:21.907269 6697 raft_consensus.cc:348] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.907383 6697 raft_consensus.cc:729] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b76761b17fc43d1b1a57866605abd56, State: Initialized, Role: LEARNER
I20240503 13:59:21.907495 6697 consensus_queue.cc:260] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 271, Last appended: 1.271, Last appended by leader: 271, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.907795 6697 ts_tablet_manager.cc:1398] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:21.908591 3628 tablet_copy_service.cc:342] P bc41cca530f54259aca15e34563208dc: Request end of tablet copy session 0b76761b17fc43d1b1a57866605abd56-97d59e7bd0da44d9a7af791f2477b183 received from {username='slave'} at 127.0.99.136:44561
I20240503 13:59:21.908692 3628 tablet_copy_service.cc:433] P bc41cca530f54259aca15e34563208dc: ending tablet copy session 0b76761b17fc43d1b1a57866605abd56-97d59e7bd0da44d9a7af791f2477b183 on tablet 97d59e7bd0da44d9a7af791f2477b183 with peer 0b76761b17fc43d1b1a57866605abd56
I20240503 13:59:21.927099 6702 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: Bootstrap replayed 1/1 log segments. Stats: ops{read=292 overwritten=0 applied=292 ignored=0} inserts{seen=961 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:21.927389 6702 tablet_bootstrap.cc:492] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: Bootstrap complete.
I20240503 13:59:21.927610 6702 ts_tablet_manager.cc:1367] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: Time spent bootstrapping tablet: real 0.203s user 0.171s sys 0.025s
I20240503 13:59:21.928048 6702 raft_consensus.cc:348] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.928185 6702 raft_consensus.cc:729] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b76761b17fc43d1b1a57866605abd56, State: Initialized, Role: LEARNER
I20240503 13:59:21.928303 6702 consensus_queue.cc:260] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 292, Last appended: 1.292, Last appended by leader: 292, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: true } }
I20240503 13:59:21.928598 6702 ts_tablet_manager.cc:1398] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:21.929332 3759 tablet_copy_service.cc:342] P 3812cd29374a4b3485883810d0774867: Request end of tablet copy session 0b76761b17fc43d1b1a57866605abd56-e967e4257bda4e4fa9b3fdfadd22368f received from {username='slave'} at 127.0.99.136:55675
I20240503 13:59:21.929435 3759 tablet_copy_service.cc:433] P 3812cd29374a4b3485883810d0774867: ending tablet copy session 0b76761b17fc43d1b1a57866605abd56-e967e4257bda4e4fa9b3fdfadd22368f on tablet e967e4257bda4e4fa9b3fdfadd22368f with peer 0b76761b17fc43d1b1a57866605abd56
I20240503 13:59:21.958923 6449 raft_consensus.cc:1203] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.257->[2.258-2.258] Dedup: 2.258->[]
I20240503 13:59:22.166867 6449 raft_consensus.cc:1203] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.302->[1.303-1.303] Dedup: 1.303->[]
I20240503 13:59:22.260353 6449 raft_consensus.cc:1203] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.291->[1.292-1.292] Dedup: 1.292->[]
I20240503 13:59:22.297783 6449 raft_consensus.cc:1203] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.270->[1.271-1.271] Dedup: 1.271->[]
I20240503 13:59:22.468417 6733 raft_consensus.cc:1050] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924: attempting to promote NON_VOTER 0b76761b17fc43d1b1a57866605abd56 to VOTER
I20240503 13:59:22.468839 6733 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 2.258, Last appended by leader: 186, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:22.470110 5106 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20240503 13:59:22.470189 6449 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 LEARNER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20240503 13:59:22.470510 6733 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240503 13:59:22.470736 6779 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240503 13:59:22.472962 6779 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Committing config change with OpId 2.259: config changed from index 258 to 259, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.473453 5106 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.474428 6449 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.476291 3463 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad reported cstate change: config changed from index 258 to 259, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.480082 5240 consensus_queue.cc:237] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 186, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:22.481129 6449 raft_consensus.cc:1261] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20240503 13:59:22.481514 6781 consensus_queue.cc:1035] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20240503 13:59:22.483124 6733 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 259 to 260, VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.483498 6449 raft_consensus.cc:2932] T 49afc417528049839897ee5600b954e8 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 259 to 260, VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.484624 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49afc417528049839897ee5600b954e8 with cas_config_opid_index 259: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:22.485782 3463 catalog_manager.cc:5484] T 49afc417528049839897ee5600b954e8 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: config changed from index 259 to 260, VOTER 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:22.488559 5087 tablet_service.cc:1514] Processing DeleteTablet for tablet 49afc417528049839897ee5600b954e8 with delete_type TABLET_DATA_TOMBSTONED (TS 6d1a97f876fc4f8698f737fcf1b782ad not found in new config with opid_index 260) from {username='slave'} at 127.0.0.1:51516
I20240503 13:59:22.489055 6793 tablet_replica.cc:330] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: stopping tablet replica
I20240503 13:59:22.489387 6793 raft_consensus.cc:2229] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.489686 6793 raft_consensus.cc:2258] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.491765 6793 ts_tablet_manager.cc:1875] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:22.493878 6793 ts_tablet_manager.cc:1888] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.259
I20240503 13:59:22.493960 6793 log.cc:1196] T 49afc417528049839897ee5600b954e8 P 6d1a97f876fc4f8698f737fcf1b782ad: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/49afc417528049839897ee5600b954e8
I20240503 13:59:22.494503 3452 catalog_manager.cc:4830] TS 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797): tablet 49afc417528049839897ee5600b954e8 (table rebalance_test_table_4 [id=1ad0716f24824080a4c3f0151947dfc9]) successfully deleted
I20240503 13:59:22.707593 6754 raft_consensus.cc:1050] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 0b76761b17fc43d1b1a57866605abd56 to VOTER
I20240503 13:59:22.708070 6754 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:22.709529 4974 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240503 13:59:22.709543 6449 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240503 13:59:22.710038 6754 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240503 13:59:22.710258 6755 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240503 13:59:22.712342 6224 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 303 to 304, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.712908 6449 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.712976 4974 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.715587 3462 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 reported cstate change: config changed from index 303 to 304, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "14a79ecc8c2c45adac20a5239cd45fd2" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39785 } attrs { promote: false replace: true } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.719487 3737 consensus_queue.cc:237] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:22.720767 6449 raft_consensus.cc:1261] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240503 13:59:22.721375 6754 consensus_queue.cc:1035] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240503 13:59:22.723109 6755 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.305: config changed from index 304 to 305, VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.723582 6449 raft_consensus.cc:2932] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 304 to 305, VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.724639 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:22.725767 3463 catalog_manager.cc:5484] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 304 to 305, VOTER 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:22.728554 4955 tablet_service.cc:1514] Processing DeleteTablet for tablet 19ebb671e56f4e29adb6a2c0a7c6b00e with delete_type TABLET_DATA_TOMBSTONED (TS 14a79ecc8c2c45adac20a5239cd45fd2 not found in new config with opid_index 305) from {username='slave'} at 127.0.0.1:33448
I20240503 13:59:22.729079 6831 tablet_replica.cc:330] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: stopping tablet replica
I20240503 13:59:22.729430 6831 raft_consensus.cc:2229] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:22.729645 6831 raft_consensus.cc:2258] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:22.731362 6831 ts_tablet_manager.cc:1875] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:22.733546 6831 ts_tablet_manager.cc:1888] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.304
I20240503 13:59:22.733631 6831 log.cc:1196] T 19ebb671e56f4e29adb6a2c0a7c6b00e P 14a79ecc8c2c45adac20a5239cd45fd2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:22.734233 3453 catalog_manager.cc:4830] TS 14a79ecc8c2c45adac20a5239cd45fd2 (127.0.99.133:39785): tablet 19ebb671e56f4e29adb6a2c0a7c6b00e (table rebalance_test_table_1 [id=e237f5fffb7742ed94e1744e85582d77]) successfully deleted
I20240503 13:59:22.781638 6224 raft_consensus.cc:1050] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: attempting to promote NON_VOTER 0b76761b17fc43d1b1a57866605abd56 to VOTER
I20240503 13:59:22.782094 6224 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:22.783336 6449 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240503 13:59:22.783346 5106 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Refusing update from remote peer 3812cd29374a4b3485883810d0774867: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240503 13:59:22.783743 6754 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240503 13:59:22.783970 6224 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240503 13:59:22.785729 6755 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 292 to 293, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.786305 5106 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.788116 6449 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.788381 3458 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 reported cstate change: config changed from index 292 to 293, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3812cd29374a4b3485883810d0774867" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:22.914260 6777 raft_consensus.cc:1050] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: attempting to promote NON_VOTER 0b76761b17fc43d1b1a57866605abd56 to VOTER
I20240503 13:59:22.914708 6777 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:22.916083 6449 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 1 LEARNER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240503 13:59:22.916157 5240 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Refusing update from remote peer bc41cca530f54259aca15e34563208dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20240503 13:59:22.916483 6856 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20240503 13:59:22.916718 6777 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20240503 13:59:22.918576 6858 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.919039 6449 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.919138 5240 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:22.921452 3462 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc reported cstate change: config changed from index 271 to 272, 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bc41cca530f54259aca15e34563208dc" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:22.971055 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f"
dest_uuid: "3812cd29374a4b3485883810d0774867"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57126
I20240503 13:59:22.971223 3737 raft_consensus.cc:595] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:22.981456 3603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183"
dest_uuid: "bc41cca530f54259aca15e34563208dc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42834
I20240503 13:59:22.981634 3603 raft_consensus.cc:595] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:23.197610 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f"
dest_uuid: "3812cd29374a4b3485883810d0774867"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57140
I20240503 13:59:23.197770 3737 raft_consensus.cc:595] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:23.208483 3603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183"
dest_uuid: "bc41cca530f54259aca15e34563208dc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42846
I20240503 13:59:23.208638 3603 raft_consensus.cc:595] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:23.311583 6754 raft_consensus.cc:979] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: : Instructing follower 0b76761b17fc43d1b1a57866605abd56 to start an election
I20240503 13:59:23.311717 6754 raft_consensus.cc:1067] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Signalling peer 0b76761b17fc43d1b1a57866605abd56 to start an election
I20240503 13:59:23.312315 6449 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f"
dest_uuid: "0b76761b17fc43d1b1a57866605abd56"
from {username='slave'} at 127.0.99.130:48413
I20240503 13:59:23.312521 6449 raft_consensus.cc:482] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:23.312645 6449 raft_consensus.cc:3037] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:23.313678 6449 raft_consensus.cc:504] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:23.314855 6449 leader_election.cc:288] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [CANDIDATE]: Term 2 election: Requested vote from peers 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107), 6d1a97f876fc4f8698f737fcf1b782ad (127.0.99.134:43797)
I20240503 13:59:23.315330 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f" candidate_uuid: "0b76761b17fc43d1b1a57866605abd56" candidate_term: 2 candidate_status { last_received { term: 1 index: 293 } } ignore_live_leader: true dest_uuid: "3812cd29374a4b3485883810d0774867"
I20240503 13:59:23.315483 3737 raft_consensus.cc:3032] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:23.315534 3737 raft_consensus.cc:729] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3812cd29374a4b3485883810d0774867, State: Running, Role: LEADER
I20240503 13:59:23.315706 3737 consensus_queue.cc:260] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 293, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:23.315940 3737 raft_consensus.cc:3037] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:23.317029 3737 raft_consensus.cc:2454] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b76761b17fc43d1b1a57866605abd56 in term 2.
I20240503 13:59:23.317427 6382 leader_election.cc:302] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b76761b17fc43d1b1a57866605abd56, 3812cd29374a4b3485883810d0774867; no voters:
I20240503 13:59:23.317705 6709 raft_consensus.cc:2781] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:23.317915 6709 raft_consensus.cc:686] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 2 LEADER]: Becoming Leader. State: Replica: 0b76761b17fc43d1b1a57866605abd56, State: Running, Role: LEADER
I20240503 13:59:23.318264 6709 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 293, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:23.320523 5106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e967e4257bda4e4fa9b3fdfadd22368f" candidate_uuid: "0b76761b17fc43d1b1a57866605abd56" candidate_term: 2 candidate_status { last_received { term: 1 index: 293 } } ignore_live_leader: true dest_uuid: "6d1a97f876fc4f8698f737fcf1b782ad"
I20240503 13:59:23.320669 5106 raft_consensus.cc:3037] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:23.321092 3458 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 reported cstate change: term changed from 1 to 2, leader changed from 3812cd29374a4b3485883810d0774867 (127.0.99.130) to 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "0b76761b17fc43d1b1a57866605abd56" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:23.321756 5106 raft_consensus.cc:2454] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b76761b17fc43d1b1a57866605abd56 in term 2.
I20240503 13:59:23.426766 3603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183"
dest_uuid: "bc41cca530f54259aca15e34563208dc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42874
I20240503 13:59:23.426937 3603 raft_consensus.cc:595] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:23.470808 6856 raft_consensus.cc:979] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: : Instructing follower 07bdad76c7bd40afb3147833b3305924 to start an election
I20240503 13:59:23.470925 6856 raft_consensus.cc:1067] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Signalling peer 07bdad76c7bd40afb3147833b3305924 to start an election
I20240503 13:59:23.471453 5240 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183"
dest_uuid: "07bdad76c7bd40afb3147833b3305924"
from {username='slave'} at 127.0.99.129:35713
I20240503 13:59:23.471601 5240 raft_consensus.cc:482] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:23.471653 5240 raft_consensus.cc:3037] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:23.472564 5240 raft_consensus.cc:504] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:23.473030 5240 leader_election.cc:288] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [CANDIDATE]: Term 2 election: Requested vote from peers bc41cca530f54259aca15e34563208dc (127.0.99.129:43281), 0b76761b17fc43d1b1a57866605abd56 (127.0.99.136:36697)
I20240503 13:59:23.473517 3603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183" candidate_uuid: "07bdad76c7bd40afb3147833b3305924" candidate_term: 2 candidate_status { last_received { term: 1 index: 272 } } ignore_live_leader: true dest_uuid: "bc41cca530f54259aca15e34563208dc"
I20240503 13:59:23.473517 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d59e7bd0da44d9a7af791f2477b183" candidate_uuid: "07bdad76c7bd40afb3147833b3305924" candidate_term: 2 candidate_status { last_received { term: 1 index: 272 } } ignore_live_leader: true dest_uuid: "0b76761b17fc43d1b1a57866605abd56"
I20240503 13:59:23.473657 3603 raft_consensus.cc:3032] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:23.473695 6449 raft_consensus.cc:3037] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:23.473707 3603 raft_consensus.cc:729] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bc41cca530f54259aca15e34563208dc, State: Running, Role: LEADER
I20240503 13:59:23.473889 3603 consensus_queue.cc:260] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 1.272, Last appended by leader: 272, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:23.474129 3603 raft_consensus.cc:3037] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:23.474833 6449 raft_consensus.cc:2454] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bdad76c7bd40afb3147833b3305924 in term 2.
I20240503 13:59:23.475123 3603 raft_consensus.cc:2454] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bdad76c7bd40afb3147833b3305924 in term 2.
I20240503 13:59:23.475284 5173 leader_election.cc:302] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bdad76c7bd40afb3147833b3305924, 0b76761b17fc43d1b1a57866605abd56; no voters:
I20240503 13:59:23.475476 6733 raft_consensus.cc:2781] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:23.475642 6733 raft_consensus.cc:686] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Becoming Leader. State: Replica: 07bdad76c7bd40afb3147833b3305924, State: Running, Role: LEADER
I20240503 13:59:23.475778 6733 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 1.272, Last appended by leader: 272, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:23.477550 3458 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 reported cstate change: term changed from 1 to 2, leader changed from bc41cca530f54259aca15e34563208dc (127.0.99.129) to 07bdad76c7bd40afb3147833b3305924 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:23.712323 3737 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Refusing update from remote peer 0b76761b17fc43d1b1a57866605abd56: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20240503 13:59:23.712980 6709 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3812cd29374a4b3485883810d0774867" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42107 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240503 13:59:23.716415 5106 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Refusing update from remote peer 0b76761b17fc43d1b1a57866605abd56: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20240503 13:59:23.716851 6709 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
W20240503 13:59:23.842867 6494 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.99.136:36697: Invalid argument: socket address is ambiguous
I20240503 13:59:23.922159 3603 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20240503 13:59:23.922617 6733 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc41cca530f54259aca15e34563208dc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43281 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20240503 13:59:23.925671 6448 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20240503 13:59:23.926362 6733 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20240503 13:59:24.327471 6448 consensus_queue.cc:237] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 293, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:24.328642 5106 raft_consensus.cc:1261] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Refusing update from remote peer 0b76761b17fc43d1b1a57866605abd56: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20240503 13:59:24.329035 6975 consensus_queue.cc:1035] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20240503 13:59:24.330724 6974 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 293 to 295, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:24.331164 5106 raft_consensus.cc:2932] T e967e4257bda4e4fa9b3fdfadd22368f P 6d1a97f876fc4f8698f737fcf1b782ad [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 293 to 295, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:24.332180 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e967e4257bda4e4fa9b3fdfadd22368f with cas_config_opid_index 293: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:24.333161 3463 catalog_manager.cc:5484] T e967e4257bda4e4fa9b3fdfadd22368f P 0b76761b17fc43d1b1a57866605abd56 reported cstate change: config changed from index 293 to 295, VOTER 3812cd29374a4b3485883810d0774867 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "0b76761b17fc43d1b1a57866605abd56" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "6d1a97f876fc4f8698f737fcf1b782ad" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43797 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:24.335978 3717 tablet_service.cc:1514] Processing DeleteTablet for tablet e967e4257bda4e4fa9b3fdfadd22368f with delete_type TABLET_DATA_TOMBSTONED (TS 3812cd29374a4b3485883810d0774867 not found in new config with opid_index 295) from {username='slave'} at 127.0.0.1:34834
I20240503 13:59:24.336367 6981 tablet_replica.cc:330] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: stopping tablet replica
I20240503 13:59:24.336501 6981 raft_consensus.cc:2229] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.336616 6981 raft_consensus.cc:2258] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.337805 6981 ts_tablet_manager.cc:1875] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:24.339834 6981 ts_tablet_manager.cc:1888] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.294
I20240503 13:59:24.339905 6981 log.cc:1196] T e967e4257bda4e4fa9b3fdfadd22368f P 3812cd29374a4b3485883810d0774867: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:24.340453 3450 catalog_manager.cc:4830] TS 3812cd29374a4b3485883810d0774867 (127.0.99.130:42107): tablet e967e4257bda4e4fa9b3fdfadd22368f (table rebalance_test_table_3 [id=eecc3a1b607145b4815145157b1b7815]) successfully deleted
I20240503 13:59:24.512974 5240 consensus_queue.cc:237] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.273, Last appended by leader: 272, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } }
I20240503 13:59:24.514101 6448 raft_consensus.cc:1261] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Refusing update from remote peer 07bdad76c7bd40afb3147833b3305924: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20240503 13:59:24.514463 7001 consensus_queue.cc:1035] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20240503 13:59:24.516053 6982 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 07bdad76c7bd40afb3147833b3305924 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 272 to 274, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:24.516456 6448 raft_consensus.cc:2932] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 272 to 274, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:24.517457 3450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97d59e7bd0da44d9a7af791f2477b183 with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:24.518719 3458 catalog_manager.cc:5484] T 97d59e7bd0da44d9a7af791f2477b183 P 0b76761b17fc43d1b1a57866605abd56 reported cstate change: config changed from index 272 to 274, VOTER bc41cca530f54259aca15e34563208dc (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "07bdad76c7bd40afb3147833b3305924" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "07bdad76c7bd40afb3147833b3305924" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43643 } attrs { promote: false } } peers { permanent_uuid: "0b76761b17fc43d1b1a57866605abd56" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 36697 } attrs { promote: false } } }
I20240503 13:59:24.521039 3587 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d59e7bd0da44d9a7af791f2477b183 with delete_type TABLET_DATA_TOMBSTONED (TS bc41cca530f54259aca15e34563208dc not found in new config with opid_index 274) from {username='slave'} at 127.0.0.1:56666
I20240503 13:59:24.521486 7002 tablet_replica.cc:330] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: stopping tablet replica
I20240503 13:59:24.521631 7002 raft_consensus.cc:2229] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:24.521750 7002 raft_consensus.cc:2258] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:24.523021 7002 ts_tablet_manager.cc:1875] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:24.524948 7002 ts_tablet_manager.cc:1888] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.273
I20240503 13:59:24.525031 7002 log.cc:1196] T 97d59e7bd0da44d9a7af791f2477b183 P bc41cca530f54259aca15e34563208dc: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:24.525581 3451 catalog_manager.cc:4830] TS bc41cca530f54259aca15e34563208dc (127.0.99.129:43281): tablet 97d59e7bd0da44d9a7af791f2477b183 (table rebalance_test_table_0 [id=d057f59ff842479da29da884b6f1612c]) successfully deleted
I20240503 13:59:24.770380 6429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.772111 5217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:24.775673 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:24.776012 3717 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:24.782766 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.794746 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.804334 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.810159 3587 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:24.892230 5217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:24.892241 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:24.898573 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.902335 3587 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:24.902457 3717 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:24.902736 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.904033 6429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:24.919942 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.018040 5217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:25.020897 4955 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.025415 5087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.031451 6429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.032791 3587 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:25.032816 3717 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:25.036238 3849 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:25.042836 4823 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
5956e2ef385b4f099bcf90b9d7156148 | 127.0.99.190:43017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:42243 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
07bdad76c7bd40afb3147833b3305924 | 127.0.99.135:43643 | HEALTHY | <none> | 2 | 0
0b76761b17fc43d1b1a57866605abd56 | 127.0.99.136:36697 | HEALTHY | <none> | 1 | 0
0c6ca23f9d9b4c91a6347c0924674bdd | 127.0.99.131:41619 | HEALTHY | <none> | 4 | 0
14a79ecc8c2c45adac20a5239cd45fd2 | 127.0.99.133:39785 | HEALTHY | <none> | 1 | 0
3812cd29374a4b3485883810d0774867 | 127.0.99.130:42107 | HEALTHY | <none> | 4 | 0
4e553631c0564b078cefd1cd1cdcc90d | 127.0.99.132:36697 | HEALTHY | <none> | 1 | 0
6d1a97f876fc4f8698f737fcf1b782ad | 127.0.99.134:43797 | HEALTHY | <none> | 1 | 0
bc41cca530f54259aca15e34563208dc | 127.0.99.129:43281 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43281
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:42107
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:41619
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:36697
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:39785
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:43797
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:43643
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:36697
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43281
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:42107
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:41619
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:36697
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:39785
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:43797
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:43643
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:36697
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:42243 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:59:25.084568 398 log_verifier.cc:126] Checking tablet 19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:25.139904 398 log_verifier.cc:177] Verified matching terms for 305 ops in tablet 19ebb671e56f4e29adb6a2c0a7c6b00e
I20240503 13:59:25.140040 398 log_verifier.cc:126] Checking tablet 2e7fa1300123438896b7be116d5560fa
I20240503 13:59:25.180290 398 log_verifier.cc:177] Verified matching terms for 220 ops in tablet 2e7fa1300123438896b7be116d5560fa
I20240503 13:59:25.180429 398 log_verifier.cc:126] Checking tablet 37a9c633962b40f1b42376cf597ad415
I20240503 13:59:25.224969 398 log_verifier.cc:177] Verified matching terms for 252 ops in tablet 37a9c633962b40f1b42376cf597ad415
I20240503 13:59:25.225104 398 log_verifier.cc:126] Checking tablet 480d84e5600d40e7b8c7fe001ba33c9f
I20240503 13:59:25.273815 398 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 480d84e5600d40e7b8c7fe001ba33c9f
I20240503 13:59:25.274014 398 log_verifier.cc:126] Checking tablet 49afc417528049839897ee5600b954e8
I20240503 13:59:25.323108 398 log_verifier.cc:177] Verified matching terms for 260 ops in tablet 49afc417528049839897ee5600b954e8
I20240503 13:59:25.323237 398 log_verifier.cc:126] Checking tablet 62fbc3876d064aa0a8e7f8f71fb3f35f
I20240503 13:59:25.372820 398 log_verifier.cc:177] Verified matching terms for 272 ops in tablet 62fbc3876d064aa0a8e7f8f71fb3f35f
I20240503 13:59:25.372984 398 log_verifier.cc:126] Checking tablet 809441335aa64f959ef084cb3c8b2434
I20240503 13:59:25.425565 398 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 809441335aa64f959ef084cb3c8b2434
I20240503 13:59:25.425696 398 log_verifier.cc:126] Checking tablet 8df17b9134914ca3a2c04800435b3697
I20240503 13:59:25.435395 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 8df17b9134914ca3a2c04800435b3697
I20240503 13:59:25.435464 398 log_verifier.cc:126] Checking tablet 97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:25.484859 398 log_verifier.cc:177] Verified matching terms for 274 ops in tablet 97d59e7bd0da44d9a7af791f2477b183
I20240503 13:59:25.485030 398 log_verifier.cc:126] Checking tablet adfa836cb94c4050bdb65e007b232f2b
I20240503 13:59:25.534230 398 log_verifier.cc:177] Verified matching terms for 284 ops in tablet adfa836cb94c4050bdb65e007b232f2b
I20240503 13:59:25.534376 398 log_verifier.cc:126] Checking tablet bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:25.576539 398 log_verifier.cc:177] Verified matching terms for 236 ops in tablet bd23523f46044bb7b7e4f44056687c43
I20240503 13:59:25.576673 398 log_verifier.cc:126] Checking tablet c1c58189e2df41c3b8b4a0924ae42a03
I20240503 13:59:25.625806 398 log_verifier.cc:177] Verified matching terms for 288 ops in tablet c1c58189e2df41c3b8b4a0924ae42a03
I20240503 13:59:25.625950 398 log_verifier.cc:126] Checking tablet dcf42adb4ddc4a25907a5c5fa94e31b7
I20240503 13:59:25.679008 398 log_verifier.cc:177] Verified matching terms for 303 ops in tablet dcf42adb4ddc4a25907a5c5fa94e31b7
I20240503 13:59:25.679159 398 log_verifier.cc:126] Checking tablet e572db34c8844384b728068168645ecc
I20240503 13:59:25.725461 398 log_verifier.cc:177] Verified matching terms for 260 ops in tablet e572db34c8844384b728068168645ecc
I20240503 13:59:25.725679 398 log_verifier.cc:126] Checking tablet e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:25.781868 398 log_verifier.cc:177] Verified matching terms for 295 ops in tablet e967e4257bda4e4fa9b3fdfadd22368f
I20240503 13:59:25.782052 398 log_verifier.cc:126] Checking tablet fc8c685a1dd4496b9be2b1e09d237371
I20240503 13:59:25.838845 398 log_verifier.cc:177] Verified matching terms for 293 ops in tablet fc8c685a1dd4496b9be2b1e09d237371
I20240503 13:59:25.840457 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 3526
I20240503 13:59:25.848520 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 3657
I20240503 13:59:25.855201 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 3788
I20240503 13:59:25.861430 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 4762
I20240503 13:59:25.866886 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 4893
I20240503 13:59:25.872493 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 5024
I20240503 13:59:25.878481 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 5155
I20240503 13:59:25.883792 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 6366
I20240503 13:59:25.888761 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 3435
2024-05-03T13:59:25Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (24533 ms)
[----------] 1 test from RebalanceParamTest (24533 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240503 13:59:25.936280 398 test_util.cc:268] Using random seed: 547844753
I20240503 13:59:25.937088 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:25.937129 398 ts_itest-base.cc:113] --------------
I20240503 13:59:25.937142 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 13:59:25.937155 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:59:25.937167 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:25Z Disabled control of system clock
I20240503 13:59:25.945961 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:45103
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:45103
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 13:59:26.055925 7127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.056200 7127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.056242 7127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.059485 7127 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:26.059587 7127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:26.059614 7127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.059654 7127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:26.059684 7127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:26.063640 7127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:45103
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.064682 7127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.065932 7127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.074029 7135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.074714 7132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.075300 7133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.076201 7127 server_base.cc:1034] running on GCE node
I20240503 13:59:26.076812 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.077906 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.079071 7127 hybrid_clock.cc:648] HybridClock initialized: now 1714744766079046 us; error 45 us; skew 500 ppm
I20240503 13:59:26.081202 7127 webserver.cc:460] Webserver started at http://127.0.99.190:40885/ using document root <none> and password file <none>
I20240503 13:59:26.081806 7127 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.081867 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.082113 7127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:26.084004 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "51c50d0b32b9449aa0d241c2b070680f"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.084523 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "51c50d0b32b9449aa0d241c2b070680f"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.088181 7127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 13:59:26.090572 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.091728 7127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:26.091856 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "51c50d0b32b9449aa0d241c2b070680f"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.091954 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.106969 7127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.107738 7127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.107910 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.114585 7127 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:45103
I20240503 13:59:26.114593 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:45103 every 8 connection(s)
I20240503 13:59:26.115639 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:26.118412 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.121621 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7127
I20240503 13:59:26.121757 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:26.124284 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f: Bootstrap starting.
I20240503 13:59:26.126935 7194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.127877 7194 log.cc:824] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:26.129736 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f: No bootstrap required, opened a new log
I20240503 13:59:26.132777 7194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c50d0b32b9449aa0d241c2b070680f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45103 } }
I20240503 13:59:26.133014 7194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.133090 7194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c50d0b32b9449aa0d241c2b070680f, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.133594 7194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c50d0b32b9449aa0d241c2b070680f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45103 } }
I20240503 13:59:26.133746 7194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:26.133797 7194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:26.133882 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.134769 7194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c50d0b32b9449aa0d241c2b070680f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45103 } }
I20240503 13:59:26.135207 7194 leader_election.cc:302] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51c50d0b32b9449aa0d241c2b070680f; no voters:
I20240503 13:59:26.135588 7194 leader_election.cc:288] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:26.135967 7199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.136159 7199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [term 1 LEADER]: Becoming Leader. State: Replica: 51c50d0b32b9449aa0d241c2b070680f, State: Running, Role: LEADER
I20240503 13:59:26.136484 7199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c50d0b32b9449aa0d241c2b070680f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45103 } }
I20240503 13:59:26.136803 7194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:26.138046 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51c50d0b32b9449aa0d241c2b070680f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c50d0b32b9449aa0d241c2b070680f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45103 } } }
I20240503 13:59:26.138182 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:26.138476 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51c50d0b32b9449aa0d241c2b070680f. Latest consensus state: current_term: 1 leader_uuid: "51c50d0b32b9449aa0d241c2b070680f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c50d0b32b9449aa0d241c2b070680f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 45103 } } }
I20240503 13:59:26.138595 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:26.138545 7208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:26.141305 7208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:26.146231 7208 catalog_manager.cc:1347] Generated new cluster ID: 09ab0964546a4b319591cb153f4d37d2
I20240503 13:59:26.146298 7208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:26.159457 7208 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:26.160567 7208 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:26.169298 7208 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 51c50d0b32b9449aa0d241c2b070680f: Generated new TSK 0
I20240503 13:59:26.170114 7208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:26.182229 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:26.295485 7218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.295768 7218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.295832 7218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.299207 7218 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:26.299294 7218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.299383 7218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:26.303340 7218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.7218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.304394 7218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.305611 7218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.314978 7223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.315491 7226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.316365 7224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.317057 7218 server_base.cc:1034] running on GCE node
I20240503 13:59:26.317526 7218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.318215 7218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.319375 7218 hybrid_clock.cc:648] HybridClock initialized: now 1714744766319357 us; error 34 us; skew 500 ppm
I20240503 13:59:26.321485 7218 webserver.cc:460] Webserver started at http://127.0.99.129:43449/ using document root <none> and password file <none>
I20240503 13:59:26.322145 7218 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.322212 7218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.322408 7218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:26.324184 7218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.324708 7218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.328398 7218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:26.330899 7232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.332036 7218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:26.332167 7218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.332269 7218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.357268 7218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.357967 7218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.358184 7218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.358753 7218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:26.359840 7218 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:26.359905 7218 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.359966 7218 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:26.359994 7218 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.368379 7218 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:41881
I20240503 13:59:26.368402 7345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:41881 every 8 connection(s)
I20240503 13:59:26.369457 7218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:26.378594 7346 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:26.378851 7346 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:26.378841 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7218
I20240503 13:59:26.378947 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:26.379468 7346 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:26.380976 7158 ts_manager.cc:194] Registered new tserver with Master: e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:26.381107 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:26.382297 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:39183
W20240503 13:59:26.493510 7349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.493795 7349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.493842 7349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.497164 7349 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:26.497251 7349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.497323 7349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:26.501207 7349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.7349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.502223 7349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.503413 7349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.512605 7355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.513381 7354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.514048 7357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.514900 7349 server_base.cc:1034] running on GCE node
I20240503 13:59:26.515332 7349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.515980 7349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.517141 7349 hybrid_clock.cc:648] HybridClock initialized: now 1714744766517119 us; error 37 us; skew 500 ppm
I20240503 13:59:26.519323 7349 webserver.cc:460] Webserver started at http://127.0.99.130:33887/ using document root <none> and password file <none>
I20240503 13:59:26.519891 7349 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.519943 7349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.520124 7349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:26.521850 7349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1e5c0b4b872d451dba60797739f83576"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.522383 7349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1e5c0b4b872d451dba60797739f83576"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.525812 7349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:26.528201 7363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.529325 7349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:26.529438 7349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1e5c0b4b872d451dba60797739f83576"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.529523 7349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.541318 7349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.541944 7349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.542157 7349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.542722 7349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:26.543653 7349 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:26.543701 7349 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.543754 7349 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:26.543782 7349 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.552014 7349 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46237
I20240503 13:59:26.552044 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46237 every 8 connection(s)
I20240503 13:59:26.553006 7349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:26.557296 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7349
I20240503 13:59:26.557404 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:26.559631 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:26.562216 7477 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:26.562453 7477 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:26.563058 7477 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:26.564132 7158 ts_manager.cc:194] Registered new tserver with Master: 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:26.564702 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:54245
W20240503 13:59:26.678016 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.678296 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.678360 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.681545 7480 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:26.681643 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.681735 7480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:26.685601 7480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.686693 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.687951 7480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.697332 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.698060 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.698581 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.699460 7480 server_base.cc:1034] running on GCE node
I20240503 13:59:26.699892 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.700489 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.701635 7480 hybrid_clock.cc:648] HybridClock initialized: now 1714744766701618 us; error 33 us; skew 500 ppm
I20240503 13:59:26.703728 7480 webserver.cc:460] Webserver started at http://127.0.99.131:44817/ using document root <none> and password file <none>
I20240503 13:59:26.704319 7480 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.704377 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.704591 7480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:26.706606 7480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9d921974718b4613b136eee9d0e1bad2"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.707113 7480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9d921974718b4613b136eee9d0e1bad2"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.710723 7480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:26.713217 7494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.714448 7480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:26.714593 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9d921974718b4613b136eee9d0e1bad2"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.714704 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.735719 7480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.736625 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.736836 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.737540 7480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:26.738574 7480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:26.738626 7480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.738683 7480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:26.738709 7480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.747104 7480 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44401
I20240503 13:59:26.747138 7607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44401 every 8 connection(s)
I20240503 13:59:26.748158 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:26.756036 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7480
I20240503 13:59:26.756201 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:26.757010 7608 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:26.757278 7608 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:26.757937 7608 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:26.758463 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:26.759119 7158 ts_manager.cc:194] Registered new tserver with Master: 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:26.759802 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:44403
W20240503 13:59:26.871284 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.871554 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.871621 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.874914 7611 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:26.875007 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.875106 7611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:26.879086 7611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.7611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.880120 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.881327 7611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.890488 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.890926 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.891775 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.892437 7611 server_base.cc:1034] running on GCE node
I20240503 13:59:26.892881 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.893579 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.894740 7611 hybrid_clock.cc:648] HybridClock initialized: now 1714744766894710 us; error 44 us; skew 500 ppm
I20240503 13:59:26.896925 7611 webserver.cc:460] Webserver started at http://127.0.99.132:39767/ using document root <none> and password file <none>
I20240503 13:59:26.897512 7611 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.897568 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.897789 7611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:26.899798 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.900285 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.903878 7611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:26.906278 7625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.907393 7611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:59:26.907516 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-6fwv"
I20240503 13:59:26.907608 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.929685 7611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.930424 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.930579 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.931110 7611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:26.932027 7611 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:26.932085 7611 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.932126 7611 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:26.932152 7611 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.941147 7611 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43133
I20240503 13:59:26.941186 7738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43133 every 8 connection(s)
I20240503 13:59:26.942207 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:26.944898 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7611
I20240503 13:59:26.945029 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:26.947335 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:26.952222 7739 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:26.952502 7739 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:26.953140 7739 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:26.954329 7158 ts_manager.cc:194] Registered new tserver with Master: 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:26.954900 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:59555
W20240503 13:59:27.060599 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.060869 7742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.060936 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.064184 7742 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:27.064278 7742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.064371 7742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:27.068295 7742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.7742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.069346 7742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.070614 7742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.079861 7750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.080248 7747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.081161 7748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.081790 7742 server_base.cc:1034] running on GCE node
I20240503 13:59:27.082288 7742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.082899 7742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.084048 7742 hybrid_clock.cc:648] HybridClock initialized: now 1714744767084033 us; error 33 us; skew 500 ppm
I20240503 13:59:27.086195 7742 webserver.cc:460] Webserver started at http://127.0.99.133:42729/ using document root <none> and password file <none>
I20240503 13:59:27.086766 7742 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.086822 7742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.087049 7742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.088877 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "476f53c7e40c4a0b86ffbd818aea818d"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.089347 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "476f53c7e40c4a0b86ffbd818aea818d"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.092862 7742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240503 13:59:27.095248 7756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.096422 7742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:27.096529 7742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "476f53c7e40c4a0b86ffbd818aea818d"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.096642 7742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.126171 7742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.126884 7742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.127053 7742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.127707 7742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.128705 7742 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.128756 7742 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.128816 7742 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.128844 7742 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.138581 7742 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36553
I20240503 13:59:27.138622 7869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36553 every 8 connection(s)
I20240503 13:59:27.139668 7742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:27.143736 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7742
I20240503 13:59:27.143860 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:27.146015 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:27.148749 7870 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:27.148980 7870 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.149576 7870 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:27.150601 7158 ts_manager.cc:194] Registered new tserver with Master: 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553)
I20240503 13:59:27.151170 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:56083
W20240503 13:59:27.256811 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.257073 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.257130 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.260365 7873 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:27.260457 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.260543 7873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:27.264436 7873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.7873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.265489 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.266738 7873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.275786 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.276543 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.277151 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.278060 7873 server_base.cc:1034] running on GCE node
I20240503 13:59:27.278527 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.279166 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.280318 7873 hybrid_clock.cc:648] HybridClock initialized: now 1714744767280298 us; error 41 us; skew 500 ppm
I20240503 13:59:27.282395 7873 webserver.cc:460] Webserver started at http://127.0.99.134:34125/ using document root <none> and password file <none>
I20240503 13:59:27.282979 7873 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.283036 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.283243 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.284986 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "708734e6729441fcb2c5952fa47771d9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.285488 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "708734e6729441fcb2c5952fa47771d9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.289141 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:59:27.291501 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.292616 7873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:27.292732 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "708734e6729441fcb2c5952fa47771d9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.292832 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.305955 7873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.306679 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.306843 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.307411 7873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.308410 7873 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.308462 7873 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.308507 7873 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.308535 7873 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.317059 7873 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36383
I20240503 13:59:27.317075 8000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36383 every 8 connection(s)
I20240503 13:59:27.318187 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:27.321862 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 7873
I20240503 13:59:27.322006 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:59:27.324170 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:27.327678 8001 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:27.327929 8001 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.328523 8001 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:27.329566 7158 ts_manager.cc:194] Registered new tserver with Master: 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383)
I20240503 13:59:27.330173 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:50363
I20240503 13:59:27.384753 7346 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
W20240503 13:59:27.436653 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.436908 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.436969 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.440199 8004 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:27.440289 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.440384 8004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:27.444132 8004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.445164 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.446413 8004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.455523 8010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.456271 8012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.456962 8009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.457790 8004 server_base.cc:1034] running on GCE node
I20240503 13:59:27.458295 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.458940 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.460098 8004 hybrid_clock.cc:648] HybridClock initialized: now 1714744767460077 us; error 39 us; skew 500 ppm
I20240503 13:59:27.462309 8004 webserver.cc:460] Webserver started at http://127.0.99.135:36633/ using document root <none> and password file <none>
I20240503 13:59:27.462903 8004 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.462966 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.463155 8004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.464901 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c5c3042421284f028fa512b7876da405"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.465409 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c5c3042421284f028fa512b7876da405"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.468935 8004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 13:59:27.471366 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.472504 8004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:27.472626 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c5c3042421284f028fa512b7876da405"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.472733 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.497422 8004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.498154 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.498318 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.498890 8004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.499886 8004 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.499939 8004 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.499985 8004 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.500015 8004 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.508327 8004 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41051
I20240503 13:59:27.508342 8131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41051 every 8 connection(s)
I20240503 13:59:27.509341 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:27.510321 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 8004
I20240503 13:59:27.510454 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:59:27.512722 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:27.520048 8132 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:27.520346 8132 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.521222 8132 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:27.522361 7158 ts_manager.cc:194] Registered new tserver with Master: c5c3042421284f028fa512b7876da405 (127.0.99.135:41051)
I20240503 13:59:27.522918 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:40623
I20240503 13:59:27.566920 7477 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
W20240503 13:59:27.627168 8135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.627440 8135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.627501 8135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.630741 8135 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:27.630831 8135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.630928 8135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:59:27.634840 8135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.8135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.635843 8135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.637107 8135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.646528 8141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.647003 8140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.647950 8143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.648566 8135 server_base.cc:1034] running on GCE node
I20240503 13:59:27.649039 8135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.649713 8135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.650868 8135 hybrid_clock.cc:648] HybridClock initialized: now 1714744767650837 us; error 47 us; skew 500 ppm
I20240503 13:59:27.652952 8135 webserver.cc:460] Webserver started at http://127.0.99.136:43353/ using document root <none> and password file <none>
I20240503 13:59:27.653540 8135 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.653597 8135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.653797 8135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.655603 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "da390a1a407a4f30b217b3d60fba563f"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.656100 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "da390a1a407a4f30b217b3d60fba563f"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.659523 8135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 13:59:27.661688 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.662737 8135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240503 13:59:27.662843 8135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "da390a1a407a4f30b217b3d60fba563f"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:27.662932 8135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.678447 8135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.679101 8135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.679272 8135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.679852 8135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.680814 8135 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.680866 8135 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.680925 8135 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.680959 8135 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.689826 8135 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43485
I20240503 13:59:27.689862 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43485 every 8 connection(s)
I20240503 13:59:27.690817 8135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:59:27.699045 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 8135
I20240503 13:59:27.699200 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:59:27.699769 8263 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:27.700047 8263 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.700726 8263 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:27.701838 7158 ts_manager.cc:194] Registered new tserver with Master: da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485)
I20240503 13:59:27.702405 7158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:44113
I20240503 13:59:27.713609 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:59:27.723851 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:59:27.723968 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7742
I20240503 13:59:27.731236 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7873
I20240503 13:59:27.739105 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 8004
I20240503 13:59:27.743849 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 8135
I20240503 13:59:27.761838 7608 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
I20240503 13:59:27.957156 7739 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
I20240503 13:59:31.500157 7158 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:31.522126 7411 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2cabe02dbf4580b53875aaa09aaffa (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.522126 7673 tablet_service.cc:1467] Processing CreateTablet for tablet a4b0fd771f844fcab38aec069b9abd00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.523236 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cabe02dbf4580b53875aaa09aaffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.523236 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0fd771f844fcab38aec069b9abd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.524202 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 3622884b9b314ccca24422d8fa05360a (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.525094 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622884b9b314ccca24422d8fa05360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.526289 7279 tablet_service.cc:1467] Processing CreateTablet for tablet a4b0fd771f844fcab38aec069b9abd00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.526305 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 3622884b9b314ccca24422d8fa05360a (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.526582 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622884b9b314ccca24422d8fa05360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.526582 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0fd771f844fcab38aec069b9abd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.526587 7410 tablet_service.cc:1467] Processing CreateTablet for tablet a4b0fd771f844fcab38aec069b9abd00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.526832 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0fd771f844fcab38aec069b9abd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.526289 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2cabe02dbf4580b53875aaa09aaffa (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.529405 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cabe02dbf4580b53875aaa09aaffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.530658 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2cabe02dbf4580b53875aaa09aaffa (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.531672 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cabe02dbf4580b53875aaa09aaffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.534715 8284 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:31.534694 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 3622884b9b314ccca24422d8fa05360a (DEFAULT_TABLE table=rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:31.534968 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622884b9b314ccca24422d8fa05360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:31.536938 8284 tablet_bootstrap.cc:654] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.537331 8283 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:31.537860 8284 log.cc:824] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:31.538615 8285 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:31.539685 8283 tablet_bootstrap.cc:654] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.540371 8285 tablet_bootstrap.cc:654] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.540664 8287 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:31.540719 8283 log.cc:824] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:31.541113 8285 log.cc:824] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:31.542670 8283 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:31.542670 8284 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:31.542698 8287 tablet_bootstrap.cc:654] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.542768 8285 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:31.542852 8283 ts_tablet_manager.cc:1367] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:59:31.542852 8284 ts_tablet_manager.cc:1367] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 13:59:31.542932 8285 ts_tablet_manager.cc:1367] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:31.543429 8287 log.cc:824] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:31.545150 8287 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:31.545331 8287 ts_tablet_manager.cc:1367] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:31.546685 8284 raft_consensus.cc:348] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.546684 8283 raft_consensus.cc:348] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.546898 8283 raft_consensus.cc:374] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.546898 8284 raft_consensus.cc:374] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.546769 8285 raft_consensus.cc:348] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.546947 8284 raft_consensus.cc:729] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.546953 8283 raft_consensus.cc:729] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.546976 8285 raft_consensus.cc:374] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.547029 8285 raft_consensus.cc:729] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.547402 8284 consensus_queue.cc:260] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.547552 8285 consensus_queue.cc:260] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.548363 8284 ts_tablet_manager.cc:1398] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:59:31.548420 8285 ts_tablet_manager.cc:1398] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:59:31.548782 8285 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:31.549127 8287 raft_consensus.cc:348] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.549381 8287 raft_consensus.cc:374] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.549448 8287 raft_consensus.cc:729] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.549680 8284 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:31.549950 8285 tablet_bootstrap.cc:654] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.549919 8287 consensus_queue.cc:260] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.550796 8284 tablet_bootstrap.cc:654] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.550863 8287 ts_tablet_manager.cc:1398] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:59:31.551429 8285 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:31.551539 8285 ts_tablet_manager.cc:1367] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:31.552138 8285 raft_consensus.cc:348] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.552335 8285 raft_consensus.cc:374] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.552414 8285 raft_consensus.cc:729] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.552623 8285 consensus_queue.cc:260] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.552973 8285 ts_tablet_manager.cc:1398] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:31.553052 8287 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:31.553218 8285 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:31.554029 8287 tablet_bootstrap.cc:654] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.554071 8285 tablet_bootstrap.cc:654] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.555243 8285 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:31.555379 8285 ts_tablet_manager.cc:1367] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:31.555575 8284 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:31.555709 8284 ts_tablet_manager.cc:1367] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:31.555887 8285 raft_consensus.cc:348] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.556036 8285 raft_consensus.cc:374] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.556113 8285 raft_consensus.cc:729] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.556285 8285 consensus_queue.cc:260] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.556317 8284 raft_consensus.cc:348] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.556464 8284 raft_consensus.cc:374] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.556535 8284 raft_consensus.cc:729] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.556645 8285 ts_tablet_manager.cc:1398] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.556754 8284 consensus_queue.cc:260] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.557168 8284 ts_tablet_manager.cc:1398] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.557230 8283 consensus_queue.cc:260] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.558295 8283 ts_tablet_manager.cc:1398] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240503 13:59:31.558508 8287 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:31.558642 8287 ts_tablet_manager.cc:1367] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:31.558729 8283 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:31.559266 8287 raft_consensus.cc:348] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.559422 8287 raft_consensus.cc:374] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.559486 8287 raft_consensus.cc:729] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.559643 8287 consensus_queue.cc:260] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.559829 8283 tablet_bootstrap.cc:654] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:31.559964 8287 ts_tablet_manager.cc:1398] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:31.561008 8283 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:31.561095 8283 ts_tablet_manager.cc:1367] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:31.561702 8283 raft_consensus.cc:348] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.561803 8283 raft_consensus.cc:374] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:31.561830 8283 raft_consensus.cc:729] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:31.561944 8283 consensus_queue.cc:260] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.562232 8283 ts_tablet_manager.cc:1398] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 13:59:31.624673 7347 tablet.cc:2369] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:31.654247 8292 raft_consensus.cc:482] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.654448 8292 raft_consensus.cc:504] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.655558 8292 leader_election.cc:288] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:31.659809 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3622884b9b314ccca24422d8fa05360a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:31.659808 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3622884b9b314ccca24422d8fa05360a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:31.660162 7693 raft_consensus.cc:2454] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:31.660162 7562 raft_consensus.cc:2454] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:31.660670 7233 leader_election.cc:302] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:31.660951 8292 raft_consensus.cc:2781] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.661034 8292 raft_consensus.cc:482] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.661077 8292 raft_consensus.cc:3037] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.661964 8292 raft_consensus.cc:504] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.662375 8292 leader_election.cc:288] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:31.662781 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3622884b9b314ccca24422d8fa05360a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:31.662818 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3622884b9b314ccca24422d8fa05360a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:31.662907 7693 raft_consensus.cc:3037] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.662981 7562 raft_consensus.cc:3037] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.663810 7562 raft_consensus.cc:2454] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:31.663829 7693 raft_consensus.cc:2454] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:31.664145 7234 leader_election.cc:302] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:31.664315 8292 raft_consensus.cc:2781] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.664543 8292 raft_consensus.cc:686] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:31.664980 8292 consensus_queue.cc:237] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:31.668123 7158 catalog_manager.cc:5484] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:31.679266 8292 raft_consensus.cc:482] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.679410 8292 raft_consensus.cc:504] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.679898 8292 leader_election.cc:288] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:31.680400 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0fd771f844fcab38aec069b9abd00" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:31.680570 7693 raft_consensus.cc:2454] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:31.680938 7234 leader_election.cc:302] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:31.681133 8292 raft_consensus.cc:2781] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.681201 8292 raft_consensus.cc:482] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.681242 8292 raft_consensus.cc:3037] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.682188 8292 raft_consensus.cc:504] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.682580 8292 leader_election.cc:288] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:31.682969 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0fd771f844fcab38aec069b9abd00" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:31.683107 7693 raft_consensus.cc:3037] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.683912 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0fd771f844fcab38aec069b9abd00" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:31.683921 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0fd771f844fcab38aec069b9abd00" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:31.684146 7693 raft_consensus.cc:2454] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:31.684223 7431 raft_consensus.cc:2454] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:31.684743 7234 leader_election.cc:302] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters: 1e5c0b4b872d451dba60797739f83576
I20240503 13:59:31.684944 8292 raft_consensus.cc:2781] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.685035 8292 raft_consensus.cc:686] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:31.685179 8292 consensus_queue.cc:237] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:31.686822 7158 catalog_manager.cc:5484] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:31.696835 7740 tablet.cc:2369] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:31.752833 7609 tablet.cc:2369] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:31.788327 8298 raft_consensus.cc:482] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:31.788502 8298 raft_consensus.cc:504] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.789631 8298 leader_election.cc:288] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:31.793542 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cabe02dbf4580b53875aaa09aaffa" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:31.793622 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cabe02dbf4580b53875aaa09aaffa" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:31.793699 7562 raft_consensus.cc:2454] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:31.793864 7300 raft_consensus.cc:2454] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:31.794091 7364 leader_election.cc:302] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:31.794344 8298 raft_consensus.cc:2781] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:31.794416 8298 raft_consensus.cc:482] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:31.794445 8298 raft_consensus.cc:3037] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.795354 8298 raft_consensus.cc:504] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.795691 8298 leader_election.cc:288] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:31.796106 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cabe02dbf4580b53875aaa09aaffa" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:31.796137 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2cabe02dbf4580b53875aaa09aaffa" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:31.796234 7300 raft_consensus.cc:3037] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.796241 7562 raft_consensus.cc:3037] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:31.797009 7562 raft_consensus.cc:2454] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:31.797066 7300 raft_consensus.cc:2454] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:31.797292 7364 leader_election.cc:302] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:31.797464 8298 raft_consensus.cc:2781] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:31.797673 8298 raft_consensus.cc:686] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:31.798063 8298 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:31.801080 7158 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:31.808954 7478 tablet.cc:2369] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:31.926479 8309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:31.926795 8309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:31.930166 8309 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 13:59:32.033442 7562 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.033465 7300 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.034384 8298 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.034823 7562 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.034854 8298 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.035349 8302 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.036764 7693 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.041119 8292 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.046116 8338 mvcc.cc:205] Tried to move back new op lower bound from 7023594586271617024 to 7023594585287540736. Current Snapshot: MvccSnapshot[applied={T|T < 7023594586240569344}]
I20240503 13:59:32.048856 7430 raft_consensus.cc:3037] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.049180 7692 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.049769 8302 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.050607 7430 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.052543 8292 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.053624 8339 mvcc.cc:205] Tried to move back new op lower bound from 7023594586246119424 to 7023594584742428672. Current Snapshot: MvccSnapshot[applied={T|T < 7023594586246119424}]
I20240503 13:59:32.073181 8343 mvcc.cc:205] Tried to move back new op lower bound from 7023594586246119424 to 7023594584742428672. Current Snapshot: MvccSnapshot[applied={T|T < 7023594586246119424}]
I20240503 13:59:32.081859 8353 mvcc.cc:205] Tried to move back new op lower bound from 7023594586310008832 to 7023594584823816192. Current Snapshot: MvccSnapshot[applied={T|T < 7023594586246168576}]
INSERT report
rows total: 2000
time total: 133.369 ms
time per row: 0.0666843 ms
I20240503 13:59:32.109433 7156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:32.115309 7411 tablet_service.cc:1467] Processing CreateTablet for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.115530 7410 tablet_service.cc:1467] Processing CreateTablet for tablet fca185e6c04648c5aa4f4ad51b0e1a39 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.115648 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9445bc10fdcf4e9b8d9167ba243bfda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.115764 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.116017 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9445bc10fdcf4e9b8d9167ba243bfda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.116360 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.116570 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aff688eaad64d149bdc1c8ab12f4ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.118280 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.118566 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9445bc10fdcf4e9b8d9167ba243bfda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.118803 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca185e6c04648c5aa4f4ad51b0e1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.118762 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.119025 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aff688eaad64d149bdc1c8ab12f4ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.120934 7672 tablet_service.cc:1467] Processing CreateTablet for tablet fca185e6c04648c5aa4f4ad51b0e1a39 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.121183 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca185e6c04648c5aa4f4ad51b0e1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.121418 8367 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:32.121583 8369 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:32.122303 8367 tablet_bootstrap.cc:654] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.122524 8369 tablet_bootstrap.cc:654] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.123335 8367 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:32.123430 8367 ts_tablet_manager.cc:1367] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:32.123572 8369 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:32.123665 8369 ts_tablet_manager.cc:1367] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:32.124111 8369 raft_consensus.cc:348] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.124130 8367 raft_consensus.cc:348] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.124238 8367 raft_consensus.cc:374] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.124243 8369 raft_consensus.cc:374] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.124188 7542 tablet_service.cc:1467] Processing CreateTablet for tablet fca185e6c04648c5aa4f4ad51b0e1a39 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.124274 8369 raft_consensus.cc:729] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.124274 8367 raft_consensus.cc:729] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.124397 8369 consensus_queue.cc:260] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.124394 8367 consensus_queue.cc:260] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.124450 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca185e6c04648c5aa4f4ad51b0e1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.124656 8369 ts_tablet_manager.cc:1398] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:32.124660 8367 ts_tablet_manager.cc:1398] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.124805 8367 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:32.124845 8369 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:32.125643 8367 tablet_bootstrap.cc:654] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.125682 8369 tablet_bootstrap.cc:654] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.126478 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.126727 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aff688eaad64d149bdc1c8ab12f4ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.127436 8370 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:32.127741 8367 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:32.127830 8367 ts_tablet_manager.cc:1367] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:32.128250 8367 raft_consensus.cc:348] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.128360 8367 raft_consensus.cc:374] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.128393 8367 raft_consensus.cc:729] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.128398 8370 tablet_bootstrap.cc:654] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.128517 8367 consensus_queue.cc:260] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.128775 8367 ts_tablet_manager.cc:1398] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:32.128930 8367 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:32.129771 8367 tablet_bootstrap.cc:654] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.130990 8367 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:32.131093 8367 ts_tablet_manager.cc:1367] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:32.131212 8368 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:32.131517 8367 raft_consensus.cc:348] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.131620 8367 raft_consensus.cc:374] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.131655 8367 raft_consensus.cc:729] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.131778 8367 consensus_queue.cc:260] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.132066 8367 ts_tablet_manager.cc:1398] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.132197 8368 tablet_bootstrap.cc:654] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.132802 8369 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:32.132892 8369 ts_tablet_manager.cc:1367] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:32.133219 8368 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:32.133311 8368 ts_tablet_manager.cc:1367] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:32.133427 8369 raft_consensus.cc:348] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.133556 8369 raft_consensus.cc:374] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.133594 8369 raft_consensus.cc:729] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.133517 8370 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:32.133721 8369 consensus_queue.cc:260] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.133767 8368 raft_consensus.cc:348] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.133867 8368 raft_consensus.cc:374] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.133904 8368 raft_consensus.cc:729] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.134050 8369 ts_tablet_manager.cc:1398] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:32.134042 8368 consensus_queue.cc:260] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.134290 8368 ts_tablet_manager.cc:1398] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:32.134457 8368 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:32.134487 8370 ts_tablet_manager.cc:1367] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:32.134986 8370 raft_consensus.cc:348] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.135107 8370 raft_consensus.cc:374] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.135147 8370 raft_consensus.cc:729] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.135325 8368 tablet_bootstrap.cc:654] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.135291 8370 consensus_queue.cc:260] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.135561 8370 ts_tablet_manager.cc:1398] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:32.137028 8368 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:32.137120 8368 ts_tablet_manager.cc:1367] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:32.137586 8368 raft_consensus.cc:348] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.137717 8368 raft_consensus.cc:374] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.137754 8368 raft_consensus.cc:729] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.137878 8368 consensus_queue.cc:260] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.138180 8368 ts_tablet_manager.cc:1398] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:32.138332 8368 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:32.139405 8368 tablet_bootstrap.cc:654] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.140389 8368 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:32.140470 8368 ts_tablet_manager.cc:1367] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:32.140834 8368 raft_consensus.cc:348] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.140926 8368 raft_consensus.cc:374] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.140954 8368 raft_consensus.cc:729] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.141073 8368 consensus_queue.cc:260] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.141323 8368 ts_tablet_manager.cc:1398] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:32.180105 8291 raft_consensus.cc:482] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.180270 8291 raft_consensus.cc:504] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.181385 8291 leader_election.cc:288] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:32.185237 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9445bc10fdcf4e9b8d9167ba243bfda5" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:32.185263 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9445bc10fdcf4e9b8d9167ba243bfda5" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:32.185411 7561 raft_consensus.cc:2454] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:32.185412 7430 raft_consensus.cc:2454] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:32.185773 7627 leader_election.cc:302] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:32.186076 8291 raft_consensus.cc:2781] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.186134 8291 raft_consensus.cc:482] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.186161 8291 raft_consensus.cc:3037] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.187007 8291 raft_consensus.cc:504] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.187355 8291 leader_election.cc:288] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:32.187688 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9445bc10fdcf4e9b8d9167ba243bfda5" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:32.187847 7561 raft_consensus.cc:3037] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.187865 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9445bc10fdcf4e9b8d9167ba243bfda5" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:32.187992 7430 raft_consensus.cc:3037] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.188695 7561 raft_consensus.cc:2454] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:32.188779 7430 raft_consensus.cc:2454] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:32.189023 7626 leader_election.cc:302] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.189206 8291 raft_consensus.cc:2781] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.189421 8291 raft_consensus.cc:686] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:32.189760 8291 consensus_queue.cc:237] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.192184 7156 catalog_manager.cc:5484] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:32.260725 8357 raft_consensus.cc:482] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.260874 8357 raft_consensus.cc:504] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.261399 8357 leader_election.cc:288] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:32.261870 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca185e6c04648c5aa4f4ad51b0e1a39" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:32.262058 7561 raft_consensus.cc:2454] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:32.262374 7364 leader_election.cc:302] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.262591 8357 raft_consensus.cc:2781] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.262651 8357 raft_consensus.cc:482] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.262683 8357 raft_consensus.cc:3037] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.263571 8357 raft_consensus.cc:504] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.263957 8357 leader_election.cc:288] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:32.264290 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca185e6c04648c5aa4f4ad51b0e1a39" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:32.264413 7561 raft_consensus.cc:3037] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.265344 7561 raft_consensus.cc:2454] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:32.265381 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca185e6c04648c5aa4f4ad51b0e1a39" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:32.265429 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca185e6c04648c5aa4f4ad51b0e1a39" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:32.265545 7693 raft_consensus.cc:2454] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:32.265628 7364 leader_election.cc:302] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.265833 8357 raft_consensus.cc:2781] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.265940 8357 raft_consensus.cc:686] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:32.266114 8357 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.267822 7156 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:32.419080 8293 raft_consensus.cc:482] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.419237 8293 raft_consensus.cc:504] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.420347 8293 leader_election.cc:288] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:32.424072 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aff688eaad64d149bdc1c8ab12f4ca1" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:32.424269 7430 raft_consensus.cc:2454] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:32.424346 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aff688eaad64d149bdc1c8ab12f4ca1" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:32.424520 7300 raft_consensus.cc:2454] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:32.424618 7496 leader_election.cc:302] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.424881 8293 raft_consensus.cc:2781] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.424943 8293 raft_consensus.cc:482] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.424985 8293 raft_consensus.cc:3037] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.425818 8293 raft_consensus.cc:504] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.426209 8293 leader_election.cc:288] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:32.426559 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aff688eaad64d149bdc1c8ab12f4ca1" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:32.426573 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aff688eaad64d149bdc1c8ab12f4ca1" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:32.426683 7300 raft_consensus.cc:3037] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.426680 7430 raft_consensus.cc:3037] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.427461 7300 raft_consensus.cc:2454] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:32.427464 7430 raft_consensus.cc:2454] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:32.427790 7496 leader_election.cc:302] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.428025 8293 raft_consensus.cc:2781] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.428241 8293 raft_consensus.cc:686] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:32.428539 8293 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:32.430953 7156 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:32.609406 8390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:32.609694 8390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:32.613027 8390 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:59:32.647136 7692 raft_consensus.cc:3037] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.648855 8357 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.652627 8298 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.668110 8291 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.673759 8291 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.705024 7430 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.705598 8293 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:32.712944 7300 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:32.714258 8293 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.327 ms
time per row: 0.0536633 ms
I20240503 13:59:32.767486 7152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:32.773661 7672 tablet_service.cc:1467] Processing CreateTablet for tablet ff246d9b2c594282bf2e99d2fd2b617d (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.773838 7410 tablet_service.cc:1467] Processing CreateTablet for tablet ff246d9b2c594282bf2e99d2fd2b617d (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.774062 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff246d9b2c594282bf2e99d2fd2b617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.774111 7279 tablet_service.cc:1467] Processing CreateTablet for tablet b23f9eeeb4af45b7b679b70514e247b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.774291 7280 tablet_service.cc:1467] Processing CreateTablet for tablet ff246d9b2c594282bf2e99d2fd2b617d (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.774363 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23f9eeeb4af45b7b679b70514e247b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.774535 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff246d9b2c594282bf2e99d2fd2b617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.774585 7278 tablet_service.cc:1467] Processing CreateTablet for tablet c74f828c28c64097ab4bf5ec66459d2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.775112 7541 tablet_service.cc:1467] Processing CreateTablet for tablet b23f9eeeb4af45b7b679b70514e247b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.775393 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23f9eeeb4af45b7b679b70514e247b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.776610 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff246d9b2c594282bf2e99d2fd2b617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.777410 8454 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:32.778208 8454 tablet_bootstrap.cc:654] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.778818 7672 tablet_service.cc:1467] Processing CreateTablet for tablet b23f9eeeb4af45b7b679b70514e247b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.778856 7673 tablet_service.cc:1467] Processing CreateTablet for tablet c74f828c28c64097ab4bf5ec66459d2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.779042 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23f9eeeb4af45b7b679b70514e247b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.779174 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74f828c28c64097ab4bf5ec66459d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.780817 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74f828c28c64097ab4bf5ec66459d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.778535 8455 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:32.778811 7410 tablet_service.cc:1467] Processing CreateTablet for tablet c74f828c28c64097ab4bf5ec66459d2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:32.781489 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74f828c28c64097ab4bf5ec66459d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:32.782230 8455 tablet_bootstrap.cc:654] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.782742 8453 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:32.783437 8455 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:32.783528 8455 ts_tablet_manager.cc:1367] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:32.783833 8453 tablet_bootstrap.cc:654] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.783984 8454 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:32.784056 8454 ts_tablet_manager.cc:1367] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:59:32.784019 8455 raft_consensus.cc:348] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.784118 8455 raft_consensus.cc:374] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.784150 8455 raft_consensus.cc:729] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.784283 8455 consensus_queue.cc:260] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.784457 8454 raft_consensus.cc:348] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.784546 8455 ts_tablet_manager.cc:1398] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:32.784574 8454 raft_consensus.cc:374] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.784603 8454 raft_consensus.cc:729] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.784682 8455 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:32.784726 8454 consensus_queue.cc:260] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.784987 8454 ts_tablet_manager.cc:1398] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:32.785470 8455 tablet_bootstrap.cc:654] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.786787 8457 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:32.786954 8455 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:32.787024 8455 ts_tablet_manager.cc:1367] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:32.787427 8455 raft_consensus.cc:348] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.787531 8455 raft_consensus.cc:374] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.787565 8455 raft_consensus.cc:729] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.787628 8457 tablet_bootstrap.cc:654] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.787750 8453 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:32.787684 8455 consensus_queue.cc:260] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.787832 8453 ts_tablet_manager.cc:1367] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:32.787972 8455 ts_tablet_manager.cc:1398] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:32.788265 8453 raft_consensus.cc:348] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.788381 8453 raft_consensus.cc:374] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.788414 8453 raft_consensus.cc:729] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.788542 8453 consensus_queue.cc:260] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.788614 8457 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:32.788684 8457 ts_tablet_manager.cc:1367] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:32.789106 8457 raft_consensus.cc:348] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.789209 8457 raft_consensus.cc:374] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.789234 8453 ts_tablet_manager.cc:1398] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.789239 8457 raft_consensus.cc:729] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.789388 8453 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:32.789348 8457 consensus_queue.cc:260] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.789602 8457 ts_tablet_manager.cc:1398] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.789770 8457 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:32.790527 8457 tablet_bootstrap.cc:654] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.791481 8453 tablet_bootstrap.cc:654] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.791597 8457 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:32.791671 8457 ts_tablet_manager.cc:1367] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:32.792074 8457 raft_consensus.cc:348] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.792186 8457 raft_consensus.cc:374] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.792232 8457 raft_consensus.cc:729] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.792359 8457 consensus_queue.cc:260] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.792659 8457 ts_tablet_manager.cc:1398] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.792812 8457 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:32.792932 8453 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:32.793007 8453 ts_tablet_manager.cc:1367] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:32.793687 8457 tablet_bootstrap.cc:654] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.794086 8453 raft_consensus.cc:348] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.794200 8453 raft_consensus.cc:374] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.794242 8453 raft_consensus.cc:729] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.794361 8453 consensus_queue.cc:260] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.794644 8453 ts_tablet_manager.cc:1398] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:32.794807 8453 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:32.795032 8457 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:32.795111 8457 ts_tablet_manager.cc:1367] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:32.795465 8457 raft_consensus.cc:348] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.795593 8457 raft_consensus.cc:374] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.795634 8457 raft_consensus.cc:729] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.795693 8453 tablet_bootstrap.cc:654] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:32.795755 8457 consensus_queue.cc:260] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.796439 8457 ts_tablet_manager.cc:1398] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:32.797221 8453 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:32.797307 8453 ts_tablet_manager.cc:1367] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:32.797665 8453 raft_consensus.cc:348] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.797770 8453 raft_consensus.cc:374] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:32.797798 8453 raft_consensus.cc:729] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:32.797915 8453 consensus_queue.cc:260] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.798183 8453 ts_tablet_manager.cc:1398] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:32.862746 8382 raft_consensus.cc:482] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.862895 8382 raft_consensus.cc:504] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.863426 8382 leader_election.cc:288] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:32.863962 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:32.864147 7430 raft_consensus.cc:2454] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:32.864513 7627 leader_election.cc:302] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:32.864717 8382 raft_consensus.cc:2781] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.864790 8382 raft_consensus.cc:482] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.864817 8382 raft_consensus.cc:3037] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.865691 8382 raft_consensus.cc:504] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.866094 8382 leader_election.cc:288] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:32.866585 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:32.866752 7430 raft_consensus.cc:3037] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.867434 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:32.867580 7300 raft_consensus.cc:2454] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:32.867550 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:32.867663 7430 raft_consensus.cc:2454] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:32.867729 7299 raft_consensus.cc:3037] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.868010 7627 leader_election.cc:302] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:32.868183 8382 raft_consensus.cc:2781] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.868255 8382 raft_consensus.cc:686] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:32.868420 8382 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.868727 7299 raft_consensus.cc:2454] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:32.870218 7154 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:32.938314 8382 raft_consensus.cc:482] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.938477 8382 raft_consensus.cc:504] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.938925 8382 leader_election.cc:288] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:32.939404 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23f9eeeb4af45b7b679b70514e247b8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:32.939497 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23f9eeeb4af45b7b679b70514e247b8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:32.939589 7562 raft_consensus.cc:2454] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:32.939635 7299 raft_consensus.cc:2454] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:32.939932 7626 leader_election.cc:302] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.940119 8382 raft_consensus.cc:2781] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.940181 8382 raft_consensus.cc:482] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.940230 8382 raft_consensus.cc:3037] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.941072 8382 raft_consensus.cc:504] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.941439 8382 leader_election.cc:288] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:32.941918 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23f9eeeb4af45b7b679b70514e247b8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:32.942085 7562 raft_consensus.cc:3037] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.943018 7562 raft_consensus.cc:2454] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:32.943307 7626 leader_election.cc:302] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:32.943456 8382 raft_consensus.cc:2781] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.943575 8382 raft_consensus.cc:686] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:32.943715 8382 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:32.946139 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23f9eeeb4af45b7b679b70514e247b8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:32.946276 7299 raft_consensus.cc:3037] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.946157 7154 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:32.947149 7299 raft_consensus.cc:2454] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:32.951973 8298 raft_consensus.cc:482] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:32.952128 8298 raft_consensus.cc:504] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.952605 8298 leader_election.cc:288] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:32.953013 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74f828c28c64097ab4bf5ec66459d2f" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:32.953155 7692 raft_consensus.cc:2454] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:32.953416 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74f828c28c64097ab4bf5ec66459d2f" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:32.953449 7365 leader_election.cc:302] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:32.953559 7299 raft_consensus.cc:2454] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:32.953737 8298 raft_consensus.cc:2781] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:32.953788 8298 raft_consensus.cc:482] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:32.953825 8298 raft_consensus.cc:3037] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.954859 8298 raft_consensus.cc:504] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.955241 8298 leader_election.cc:288] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:32.955587 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74f828c28c64097ab4bf5ec66459d2f" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:32.955711 7299 raft_consensus.cc:3037] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.955753 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74f828c28c64097ab4bf5ec66459d2f" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:32.955858 7692 raft_consensus.cc:3037] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:32.956681 7299 raft_consensus.cc:2454] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:32.956748 7692 raft_consensus.cc:2454] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:32.957017 7364 leader_election.cc:302] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:32.957201 8298 raft_consensus.cc:2781] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:32.957307 8298 raft_consensus.cc:686] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:32.957440 8298 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:32.959313 7154 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:33.088315 8470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:33.088618 8470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:33.092008 8470 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:59:33.189738 7430 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.190021 7299 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.190332 7692 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.190395 8382 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.190734 8298 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.191399 8298 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.192054 7300 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.192451 7562 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.194088 8420 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.200994 8420 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.221453 7299 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.222596 8291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.44 ms
time per row: 0.0607198 ms
I20240503 13:59:33.260473 7152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:33.266005 7410 tablet_service.cc:1467] Processing CreateTablet for tablet f73a5db0cae647d4b3375e6d42358307 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.266073 7541 tablet_service.cc:1467] Processing CreateTablet for tablet f73a5db0cae647d4b3375e6d42358307 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.266359 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73a5db0cae647d4b3375e6d42358307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.266359 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73a5db0cae647d4b3375e6d42358307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.266520 7278 tablet_service.cc:1467] Processing CreateTablet for tablet f73a5db0cae647d4b3375e6d42358307 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.266813 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73a5db0cae647d4b3375e6d42358307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.268414 8457 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:33.268524 8454 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:33.268575 8455 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:33.269271 8457 tablet_bootstrap.cc:654] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.269271 8454 tablet_bootstrap.cc:654] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.269428 8455 tablet_bootstrap.cc:654] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.270107 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffd8718851475099a6696e4a2b8901 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.270274 8454 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:33.270337 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffd8718851475099a6696e4a2b8901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.270356 8454 ts_tablet_manager.cc:1367] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:33.270597 7672 tablet_service.cc:1467] Processing CreateTablet for tablet a1115afa5ead4c95aea379ad9987cbe0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.270766 7541 tablet_service.cc:1467] Processing CreateTablet for tablet a1115afa5ead4c95aea379ad9987cbe0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.270864 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1115afa5ead4c95aea379ad9987cbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.270802 8454 raft_consensus.cc:348] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.270908 8454 raft_consensus.cc:374] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.270937 8454 raft_consensus.cc:729] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.270980 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1115afa5ead4c95aea379ad9987cbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.271060 8454 consensus_queue.cc:260] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.271329 8454 ts_tablet_manager.cc:1398] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.271705 7278 tablet_service.cc:1467] Processing CreateTablet for tablet a1115afa5ead4c95aea379ad9987cbe0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.271931 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1115afa5ead4c95aea379ad9987cbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.272804 8455 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:33.272902 8455 ts_tablet_manager.cc:1367] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:33.273149 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffd8718851475099a6696e4a2b8901 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.273393 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffd8718851475099a6696e4a2b8901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.273375 8455 raft_consensus.cc:348] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.273483 8455 raft_consensus.cc:374] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.273514 8455 raft_consensus.cc:729] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.273628 8455 consensus_queue.cc:260] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.273910 8455 ts_tablet_manager.cc:1398] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:33.274103 8453 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:33.274055 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffd8718851475099a6696e4a2b8901 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.274137 8455 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:33.274272 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffd8718851475099a6696e4a2b8901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.274911 8453 tablet_bootstrap.cc:654] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.274994 8455 tablet_bootstrap.cc:654] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.275216 8454 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:33.276027 8454 tablet_bootstrap.cc:654] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.276598 8453 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:33.276700 8453 ts_tablet_manager.cc:1367] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:33.277102 8453 raft_consensus.cc:348] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.277204 8453 raft_consensus.cc:374] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.277235 8453 raft_consensus.cc:729] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.277258 8454 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:33.277330 8454 ts_tablet_manager.cc:1367] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:33.277328 8453 consensus_queue.cc:260] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.277558 8453 ts_tablet_manager.cc:1398] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.277671 8453 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:33.277724 8454 raft_consensus.cc:348] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.277818 8454 raft_consensus.cc:374] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.277848 8454 raft_consensus.cc:729] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.277956 8454 consensus_queue.cc:260] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.278461 8453 tablet_bootstrap.cc:654] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.278507 8454 ts_tablet_manager.cc:1398] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.278661 8454 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:33.279479 8454 tablet_bootstrap.cc:654] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.279870 8453 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:33.279939 8453 ts_tablet_manager.cc:1367] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:33.280165 8455 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:33.280243 8455 ts_tablet_manager.cc:1367] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:33.280330 8453 raft_consensus.cc:348] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.280419 8453 raft_consensus.cc:374] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.280447 8453 raft_consensus.cc:729] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.280514 8457 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:33.280560 8453 consensus_queue.cc:260] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.280602 8457 ts_tablet_manager.cc:1367] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20240503 13:59:33.280687 8455 raft_consensus.cc:348] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.280817 8453 ts_tablet_manager.cc:1398] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.280823 8455 raft_consensus.cc:374] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.280858 8455 raft_consensus.cc:729] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.280967 8455 consensus_queue.cc:260] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.281224 8455 ts_tablet_manager.cc:1398] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:33.281239 8457 raft_consensus.cc:348] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.281338 8457 raft_consensus.cc:374] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.281368 8457 raft_consensus.cc:729] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.281510 8454 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:33.281487 8457 consensus_queue.cc:260] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.281585 8454 ts_tablet_manager.cc:1367] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:33.281752 8457 ts_tablet_manager.cc:1398] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.281925 8457 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:33.282123 8454 raft_consensus.cc:348] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.282219 8454 raft_consensus.cc:374] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.282248 8454 raft_consensus.cc:729] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.282351 8454 consensus_queue.cc:260] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.282604 8454 ts_tablet_manager.cc:1398] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.283129 8457 tablet_bootstrap.cc:654] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.285152 8457 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:33.285250 8457 ts_tablet_manager.cc:1367] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:33.285666 8457 raft_consensus.cc:348] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.285782 8457 raft_consensus.cc:374] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.285815 8457 raft_consensus.cc:729] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.286386 8457 consensus_queue.cc:260] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.286742 8457 ts_tablet_manager.cc:1398] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.292330 8388 raft_consensus.cc:482] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.292454 8388 raft_consensus.cc:504] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.292882 8388 leader_election.cc:288] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:33.293306 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1115afa5ead4c95aea379ad9987cbe0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:33.293458 7298 raft_consensus.cc:2454] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:33.293764 7495 leader_election.cc:302] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.293926 8388 raft_consensus.cc:2781] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.293998 8388 raft_consensus.cc:482] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.294044 8388 raft_consensus.cc:3037] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.295094 8388 raft_consensus.cc:504] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.295464 8388 leader_election.cc:288] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:33.295861 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1115afa5ead4c95aea379ad9987cbe0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:33.295984 7298 raft_consensus.cc:3037] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.296854 7298 raft_consensus.cc:2454] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:33.297149 7495 leader_election.cc:302] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.297327 8388 raft_consensus.cc:2781] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.297456 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1115afa5ead4c95aea379ad9987cbe0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:33.297516 8388 raft_consensus.cc:686] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:33.297443 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1115afa5ead4c95aea379ad9987cbe0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:33.297595 7693 raft_consensus.cc:3037] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.297674 8388 consensus_queue.cc:237] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.298521 7693 raft_consensus.cc:2454] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:33.299451 7158 catalog_manager.cc:5484] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.304818 8414 raft_consensus.cc:482] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.304955 8414 raft_consensus.cc:504] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.305382 8414 leader_election.cc:288] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:33.305830 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffd8718851475099a6696e4a2b8901" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:33.305848 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffd8718851475099a6696e4a2b8901" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:33.306003 7693 raft_consensus.cc:2454] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:33.306026 7562 raft_consensus.cc:2454] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:33.306373 7364 leader_election.cc:302] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:33.306586 8414 raft_consensus.cc:2781] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.306653 8414 raft_consensus.cc:482] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.306713 8414 raft_consensus.cc:3037] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.307596 8414 raft_consensus.cc:504] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.307957 8414 leader_election.cc:288] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:33.308349 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffd8718851475099a6696e4a2b8901" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:33.308349 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffd8718851475099a6696e4a2b8901" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:33.308481 7693 raft_consensus.cc:3037] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.308481 7562 raft_consensus.cc:3037] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.309473 7562 raft_consensus.cc:2454] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:33.309473 7693 raft_consensus.cc:2454] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:33.309777 7364 leader_election.cc:302] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:33.310057 8414 raft_consensus.cc:2781] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.310194 8414 raft_consensus.cc:686] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:33.310344 8414 consensus_queue.cc:237] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.312181 7158 catalog_manager.cc:5484] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:33.312490 8355 raft_consensus.cc:482] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.312608 8355 raft_consensus.cc:504] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.313022 8355 leader_election.cc:288] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:33.313330 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73a5db0cae647d4b3375e6d42358307" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:33.313450 7430 raft_consensus.cc:2454] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:33.313781 7234 leader_election.cc:302] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.314056 8355 raft_consensus.cc:2781] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.314117 8355 raft_consensus.cc:482] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.314078 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73a5db0cae647d4b3375e6d42358307" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:33.314154 8355 raft_consensus.cc:3037] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.314225 7562 raft_consensus.cc:2454] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:33.315207 8355 raft_consensus.cc:504] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.315549 8355 leader_election.cc:288] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:33.315819 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73a5db0cae647d4b3375e6d42358307" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:33.315932 7430 raft_consensus.cc:3037] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.316074 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73a5db0cae647d4b3375e6d42358307" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:33.316208 7562 raft_consensus.cc:3037] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.316756 7430 raft_consensus.cc:2454] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:33.317055 7562 raft_consensus.cc:2454] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:33.317059 7234 leader_election.cc:302] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.317229 8355 raft_consensus.cc:2781] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.317332 8355 raft_consensus.cc:686] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:33.317461 8355 consensus_queue.cc:237] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.319262 7158 catalog_manager.cc:5484] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:33.438599 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:33.438889 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:33.442246 8519 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:59:33.537335 7562 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.538221 7430 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.539017 8355 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.539237 8517 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.541304 7298 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.541738 8388 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.545435 7561 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.546010 7693 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.546662 7693 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:33.547051 8514 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.547972 8414 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:33.548650 8414 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.173 ms
time per row: 0.0610865 ms
I20240503 13:59:33.611333 7151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:33.617447 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 902cf3db3581465c83170c4b13b3bffb (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.617816 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902cf3db3581465c83170c4b13b3bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.617952 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 33b7b43742e34431af0c16d320b35dd7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.618257 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.618300 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 902cf3db3581465c83170c4b13b3bffb (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.618559 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902cf3db3581465c83170c4b13b3bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.620812 8454 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:33.620839 7672 tablet_service.cc:1467] Processing CreateTablet for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.620862 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 902cf3db3581465c83170c4b13b3bffb (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.621089 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3eaa7e5f641436a8f0cfdaf83443ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.621098 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902cf3db3581465c83170c4b13b3bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.622110 8454 tablet_bootstrap.cc:654] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.623169 8457 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:33.623162 8454 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:33.623251 8454 ts_tablet_manager.cc:1367] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:33.623358 7279 tablet_service.cc:1467] Processing CreateTablet for tablet 33b7b43742e34431af0c16d320b35dd7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.623605 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.623725 8454 raft_consensus.cc:348] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.623840 8454 raft_consensus.cc:374] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.623870 8454 raft_consensus.cc:729] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.624042 8457 tablet_bootstrap.cc:654] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.623996 8454 consensus_queue.cc:260] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.624280 8454 ts_tablet_manager.cc:1398] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.625545 8453 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:33.626432 8453 tablet_bootstrap.cc:654] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.627805 8453 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:33.627908 8453 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:33.628238 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 33b7b43742e34431af0c16d320b35dd7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.628329 8453 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.628435 8453 raft_consensus.cc:374] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.628468 8453 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.628520 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.628597 8453 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.628897 8453 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.629057 8453 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:33.629938 8453 tablet_bootstrap.cc:654] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.630769 8455 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:33.631651 8455 tablet_bootstrap.cc:654] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.632159 8453 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:33.632300 8453 ts_tablet_manager.cc:1367] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.632781 8453 raft_consensus.cc:348] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.632876 8453 raft_consensus.cc:374] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.632905 8453 raft_consensus.cc:729] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.633085 8453 consensus_queue.cc:260] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.633636 8453 ts_tablet_manager.cc:1398] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.633944 7278 tablet_service.cc:1467] Processing CreateTablet for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.634267 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3eaa7e5f641436a8f0cfdaf83443ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.634363 7542 tablet_service.cc:1467] Processing CreateTablet for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:33.634642 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3eaa7e5f641436a8f0cfdaf83443ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:33.636683 8454 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:33.637398 8455 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:33.637499 8455 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:33.637630 8457 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:33.637722 8457 ts_tablet_manager.cc:1367] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:59:33.637883 8454 tablet_bootstrap.cc:654] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.638010 8455 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.638116 8455 raft_consensus.cc:374] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.638144 8455 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.638254 8455 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.638502 8455 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.638485 8457 raft_consensus.cc:348] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.638592 8457 raft_consensus.cc:374] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.638634 8457 raft_consensus.cc:729] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.638649 8455 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:33.638765 8457 consensus_queue.cc:260] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.639030 8457 ts_tablet_manager.cc:1398] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.639204 8457 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:33.639447 8454 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:33.639531 8454 ts_tablet_manager.cc:1367] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.639981 8454 raft_consensus.cc:348] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.640091 8457 tablet_bootstrap.cc:654] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.640102 8454 raft_consensus.cc:374] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.640134 8454 raft_consensus.cc:729] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.640247 8454 consensus_queue.cc:260] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.640614 8454 ts_tablet_manager.cc:1398] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.641873 8457 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:33.641968 8457 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:33.642014 8455 tablet_bootstrap.cc:654] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.642446 8457 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.642544 8457 raft_consensus.cc:374] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.642572 8457 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.642702 8457 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.643064 8457 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:33.643230 8457 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:33.643653 8455 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:33.643748 8455 ts_tablet_manager.cc:1367] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:33.644156 8457 tablet_bootstrap.cc:654] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:33.644205 8455 raft_consensus.cc:348] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.644317 8455 raft_consensus.cc:374] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.644349 8455 raft_consensus.cc:729] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.644474 8455 consensus_queue.cc:260] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.644711 8455 ts_tablet_manager.cc:1398] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:33.646435 8457 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:33.646528 8457 ts_tablet_manager.cc:1367] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:33.646975 8457 raft_consensus.cc:348] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.647086 8457 raft_consensus.cc:374] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:33.647128 8457 raft_consensus.cc:729] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:33.647238 8457 consensus_queue.cc:260] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.647482 8457 ts_tablet_manager.cc:1398] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:33.648432 8553 raft_consensus.cc:482] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.648524 8553 raft_consensus.cc:504] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.648918 8553 leader_election.cc:288] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:33.649258 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3eaa7e5f641436a8f0cfdaf83443ca1" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:33.649273 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3eaa7e5f641436a8f0cfdaf83443ca1" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:33.649396 7693 raft_consensus.cc:2454] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:33.649392 7561 raft_consensus.cc:2454] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:33.649688 7234 leader_election.cc:302] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.649834 8553 raft_consensus.cc:2781] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.649886 8553 raft_consensus.cc:482] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.649927 8553 raft_consensus.cc:3037] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.650816 8553 raft_consensus.cc:504] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.651152 8553 leader_election.cc:288] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:33.651496 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3eaa7e5f641436a8f0cfdaf83443ca1" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:33.651497 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3eaa7e5f641436a8f0cfdaf83443ca1" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:33.651615 7693 raft_consensus.cc:3037] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.651628 7561 raft_consensus.cc:3037] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.652364 7561 raft_consensus.cc:2454] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:33.652369 7693 raft_consensus.cc:2454] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:33.652665 7234 leader_election.cc:302] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.652920 8553 raft_consensus.cc:2781] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.653015 8553 raft_consensus.cc:686] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:33.653168 8553 consensus_queue.cc:237] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:33.654884 7153 catalog_manager.cc:5484] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.846647 8414 raft_consensus.cc:482] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.846794 8414 raft_consensus.cc:504] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.847251 8414 leader_election.cc:288] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:33.847604 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902cf3db3581465c83170c4b13b3bffb" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:33.847607 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902cf3db3581465c83170c4b13b3bffb" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:33.847756 7298 raft_consensus.cc:2454] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:33.847776 7561 raft_consensus.cc:2454] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:33.848006 7364 leader_election.cc:302] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:33.848186 8414 raft_consensus.cc:2781] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.848240 8414 raft_consensus.cc:482] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.848277 8414 raft_consensus.cc:3037] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.849103 8414 raft_consensus.cc:504] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.849417 8414 leader_election.cc:288] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:33.849740 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902cf3db3581465c83170c4b13b3bffb" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:33.849778 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902cf3db3581465c83170c4b13b3bffb" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:33.849855 7561 raft_consensus.cc:3037] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.849876 7298 raft_consensus.cc:3037] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.850849 7561 raft_consensus.cc:2454] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:33.850865 7298 raft_consensus.cc:2454] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:33.851106 7364 leader_election.cc:302] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:33.851292 8414 raft_consensus.cc:2781] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.851377 8414 raft_consensus.cc:686] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:33.851521 8414 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:33.853215 7153 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:33.890452 8414 raft_consensus.cc:482] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:33.890604 8414 raft_consensus.cc:504] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.891063 8414 leader_election.cc:288] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:33.891638 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:33.891631 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:33.891777 7693 raft_consensus.cc:2454] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:33.891786 7298 raft_consensus.cc:2454] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:33.892107 7365 leader_election.cc:302] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:33.892275 8414 raft_consensus.cc:2781] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:33.892336 8414 raft_consensus.cc:482] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:33.892364 8414 raft_consensus.cc:3037] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.893209 8414 raft_consensus.cc:504] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.893570 8414 leader_election.cc:288] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:33.893934 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:33.893919 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:33.894097 7298 raft_consensus.cc:3037] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.894105 7693 raft_consensus.cc:3037] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:33.894855 7693 raft_consensus.cc:2454] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:33.894865 7298 raft_consensus.cc:2454] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:33.895185 7365 leader_election.cc:302] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:33.895422 8414 raft_consensus.cc:2781] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:33.895499 8414 raft_consensus.cc:686] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:33.895646 8414 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:33.897410 7153 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:34.041321 8574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:34.041607 8574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:34.044879 8574 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 13:59:34.078752 8553 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.083065 8553 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.135428 7298 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.135982 8414 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.137022 7693 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.139032 7299 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.139451 7561 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.139724 8357 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.146745 8298 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.151482 8414 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.691 ms
time per row: 0.0523455 ms
I20240503 13:59:34.203011 7151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:34.208851 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 2212cf72b6ee4ee5b086455a75482e3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.208851 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 2212cf72b6ee4ee5b086455a75482e3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.209187 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2212cf72b6ee4ee5b086455a75482e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.209204 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2212cf72b6ee4ee5b086455a75482e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.209313 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 2212cf72b6ee4ee5b086455a75482e3b (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.209564 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2212cf72b6ee4ee5b086455a75482e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.209630 7541 tablet_service.cc:1467] Processing CreateTablet for tablet faa9fb136138439b8f694ac95dbfa2e7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.209836 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa9fb136138439b8f694ac95dbfa2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.210469 7278 tablet_service.cc:1467] Processing CreateTablet for tablet ade40ddaf0304b1a9e94db6630175c0e (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.210747 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade40ddaf0304b1a9e94db6630175c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.208851 7409 tablet_service.cc:1467] Processing CreateTablet for tablet faa9fb136138439b8f694ac95dbfa2e7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.211701 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa9fb136138439b8f694ac95dbfa2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.211647 7540 tablet_service.cc:1467] Processing CreateTablet for tablet ade40ddaf0304b1a9e94db6630175c0e (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.211885 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade40ddaf0304b1a9e94db6630175c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.212899 7410 tablet_service.cc:1467] Processing CreateTablet for tablet ade40ddaf0304b1a9e94db6630175c0e (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.213245 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade40ddaf0304b1a9e94db6630175c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.213963 8631 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:34.213929 7673 tablet_service.cc:1467] Processing CreateTablet for tablet faa9fb136138439b8f694ac95dbfa2e7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.214293 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa9fb136138439b8f694ac95dbfa2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.215052 8631 tablet_bootstrap.cc:654] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.215471 8632 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:34.216325 8631 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:34.216410 8631 ts_tablet_manager.cc:1367] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:34.216462 8632 tablet_bootstrap.cc:654] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.216804 8634 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:34.216872 8631 raft_consensus.cc:348] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.216979 8631 raft_consensus.cc:374] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.217008 8631 raft_consensus.cc:729] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.217135 8631 consensus_queue.cc:260] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.217197 8630 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:34.217424 8631 ts_tablet_manager.cc:1398] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.217586 8631 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:34.217659 8634 tablet_bootstrap.cc:654] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.218075 8630 tablet_bootstrap.cc:654] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.218413 8631 tablet_bootstrap.cc:654] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.218531 8632 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:34.218632 8632 ts_tablet_manager.cc:1367] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:34.218710 8634 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:34.218786 8634 ts_tablet_manager.cc:1367] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:34.219074 8630 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:34.219158 8630 ts_tablet_manager.cc:1367] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:34.219182 8634 raft_consensus.cc:348] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.219174 8632 raft_consensus.cc:348] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.219277 8632 raft_consensus.cc:374] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.219278 8634 raft_consensus.cc:374] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.219308 8632 raft_consensus.cc:729] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.219308 8634 raft_consensus.cc:729] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.219440 8634 consensus_queue.cc:260] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.219457 8632 consensus_queue.cc:260] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.219553 8630 raft_consensus.cc:348] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.219655 8630 raft_consensus.cc:374] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.219692 8630 raft_consensus.cc:729] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.219703 8634 ts_tablet_manager.cc:1398] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.219802 8630 consensus_queue.cc:260] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.219857 8634 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:34.220038 8630 ts_tablet_manager.cc:1398] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.220191 8630 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:34.220708 8634 tablet_bootstrap.cc:654] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.220997 8630 tablet_bootstrap.cc:654] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.222335 8634 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:34.222426 8634 ts_tablet_manager.cc:1367] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:34.222855 8634 raft_consensus.cc:348] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.222967 8634 raft_consensus.cc:374] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.222995 8634 raft_consensus.cc:729] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.223009 8631 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:34.223088 8631 ts_tablet_manager.cc:1367] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 13:59:34.223116 8634 consensus_queue.cc:260] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.223377 8634 ts_tablet_manager.cc:1398] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.223500 8631 raft_consensus.cc:348] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.223601 8631 raft_consensus.cc:374] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.223634 8631 raft_consensus.cc:729] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.223742 8631 consensus_queue.cc:260] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.223783 8632 ts_tablet_manager.cc:1398] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:34.224000 8631 ts_tablet_manager.cc:1398] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.224144 8631 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:34.224205 8630 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:34.224272 8630 ts_tablet_manager.cc:1367] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:34.224694 8630 raft_consensus.cc:348] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.224788 8630 raft_consensus.cc:374] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.224817 8630 raft_consensus.cc:729] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.224925 8630 consensus_queue.cc:260] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.225003 8631 tablet_bootstrap.cc:654] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.225185 8630 ts_tablet_manager.cc:1398] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.225342 8630 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:34.226212 8630 tablet_bootstrap.cc:654] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.226783 8631 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:34.226861 8631 ts_tablet_manager.cc:1367] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:34.227289 8631 raft_consensus.cc:348] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.227406 8631 raft_consensus.cc:374] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.227447 8631 raft_consensus.cc:729] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.227557 8631 consensus_queue.cc:260] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.227860 8631 ts_tablet_manager.cc:1398] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.227944 8630 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:34.228026 8630 ts_tablet_manager.cc:1367] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:34.228425 8630 raft_consensus.cc:348] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.228509 8630 raft_consensus.cc:374] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.228535 8630 raft_consensus.cc:729] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.228667 8630 consensus_queue.cc:260] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.228982 8630 ts_tablet_manager.cc:1398] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.235018 8556 raft_consensus.cc:482] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.235126 8556 raft_consensus.cc:504] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.235466 8556 leader_election.cc:288] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:34.235857 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa9fb136138439b8f694ac95dbfa2e7" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:34.235873 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa9fb136138439b8f694ac95dbfa2e7" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:34.235996 7430 raft_consensus.cc:2454] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:34.236001 7692 raft_consensus.cc:2454] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:34.236313 7496 leader_election.cc:302] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:34.236490 8556 raft_consensus.cc:2781] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.236546 8556 raft_consensus.cc:482] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.236599 8556 raft_consensus.cc:3037] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.237816 8556 raft_consensus.cc:504] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.238193 8556 leader_election.cc:288] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:34.238523 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa9fb136138439b8f694ac95dbfa2e7" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:34.238577 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa9fb136138439b8f694ac95dbfa2e7" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:34.238659 7430 raft_consensus.cc:3037] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.238693 7692 raft_consensus.cc:3037] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.239590 7430 raft_consensus.cc:2454] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:34.239589 7692 raft_consensus.cc:2454] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:34.239873 7496 leader_election.cc:302] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:34.240026 8556 raft_consensus.cc:2781] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.240244 8556 raft_consensus.cc:686] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:34.240424 8556 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.242143 7153 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:34.243043 8556 raft_consensus.cc:482] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.243153 8556 raft_consensus.cc:504] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.243448 8556 leader_election.cc:288] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:34.243893 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade40ddaf0304b1a9e94db6630175c0e" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:34.244029 7430 raft_consensus.cc:2454] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:34.244114 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade40ddaf0304b1a9e94db6630175c0e" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:34.244257 7298 raft_consensus.cc:2454] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:34.244513 7495 leader_election.cc:302] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:34.244663 8556 raft_consensus.cc:2781] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.244714 8556 raft_consensus.cc:482] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.244737 8556 raft_consensus.cc:3037] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.266958 8556 raft_consensus.cc:504] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.267452 8556 leader_election.cc:288] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:34.267906 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade40ddaf0304b1a9e94db6630175c0e" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:34.267904 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade40ddaf0304b1a9e94db6630175c0e" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:34.268044 7430 raft_consensus.cc:3037] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.268052 7298 raft_consensus.cc:3037] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.268936 7298 raft_consensus.cc:2454] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:34.268936 7430 raft_consensus.cc:2454] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:34.269245 7496 leader_election.cc:302] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:34.269425 8556 raft_consensus.cc:2781] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.269538 8556 raft_consensus.cc:686] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:34.269836 8556 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:34.271642 7153 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:34.335551 8298 raft_consensus.cc:482] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.335732 8298 raft_consensus.cc:504] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.336234 8298 leader_election.cc:288] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:34.336750 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2212cf72b6ee4ee5b086455a75482e3b" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:34.336750 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2212cf72b6ee4ee5b086455a75482e3b" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:34.336915 7692 raft_consensus.cc:2454] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:34.336916 7562 raft_consensus.cc:2454] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:34.337265 7365 leader_election.cc:302] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:34.337461 8298 raft_consensus.cc:2781] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.337519 8298 raft_consensus.cc:482] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.337563 8298 raft_consensus.cc:3037] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.338703 8298 raft_consensus.cc:504] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.339042 8298 leader_election.cc:288] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:34.339414 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2212cf72b6ee4ee5b086455a75482e3b" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:34.339432 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2212cf72b6ee4ee5b086455a75482e3b" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:34.339535 7562 raft_consensus.cc:3037] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.339541 7692 raft_consensus.cc:3037] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.340298 7562 raft_consensus.cc:2454] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:34.340312 7692 raft_consensus.cc:2454] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:34.340579 7365 leader_election.cc:302] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:34.340791 8298 raft_consensus.cc:2781] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.340888 8298 raft_consensus.cc:686] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:34.341040 8298 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.342772 7153 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:34.482880 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:34.483175 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:34.486445 8647 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 13:59:34.577497 7298 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.578039 8643 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.579703 7430 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.581337 7692 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.581828 8298 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.584975 7430 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.585429 8643 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.585793 8643 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.589322 7693 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.594655 7562 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.595980 8414 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.598368 8676 mvcc.cc:205] Tried to move back new op lower bound from 7023594596657483776 to 7023594595702456320. Current Snapshot: MvccSnapshot[applied={T|T < 7023594596657483776}]
I20240503 13:59:34.598851 8679 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.51 ms
time per row: 0.0627551 ms
I20240503 13:59:34.659263 7157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:34.665041 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 9b04204ad281452794d724fb4d7212f8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.665036 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 9b04204ad281452794d724fb4d7212f8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.665390 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b04204ad281452794d724fb4d7212f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.665395 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b04204ad281452794d724fb4d7212f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.665601 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 9b04204ad281452794d724fb4d7212f8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.665882 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b04204ad281452794d724fb4d7212f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.666364 7279 tablet_service.cc:1467] Processing CreateTablet for tablet 86634d81241c44cebcb602a6982f4244 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.666595 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86634d81241c44cebcb602a6982f4244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.666658 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 86634d81241c44cebcb602a6982f4244 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.666740 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 86634d81241c44cebcb602a6982f4244 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.666922 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86634d81241c44cebcb602a6982f4244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.666963 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86634d81241c44cebcb602a6982f4244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.667029 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 4582c127b7d34e08a68ea9094de11f19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.667227 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4582c127b7d34e08a68ea9094de11f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.667434 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 4582c127b7d34e08a68ea9094de11f19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.667631 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4582c127b7d34e08a68ea9094de11f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.667717 8631 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:34.668474 8631 tablet_bootstrap.cc:654] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.668769 8630 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:34.669277 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 4582c127b7d34e08a68ea9094de11f19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.669507 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4582c127b7d34e08a68ea9094de11f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.669543 8631 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:34.669631 8630 tablet_bootstrap.cc:654] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.669651 8631 ts_tablet_manager.cc:1367] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:34.670166 8631 raft_consensus.cc:348] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.670271 8631 raft_consensus.cc:374] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.670301 8631 raft_consensus.cc:729] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.670442 8631 consensus_queue.cc:260] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.670738 8631 ts_tablet_manager.cc:1398] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.671416 8631 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:34.671567 8630 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:34.671665 8630 ts_tablet_manager.cc:1367] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:34.672195 8630 raft_consensus.cc:348] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.672307 8630 raft_consensus.cc:374] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.672338 8630 raft_consensus.cc:729] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.672525 8631 tablet_bootstrap.cc:654] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.672477 8630 consensus_queue.cc:260] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.672773 8630 ts_tablet_manager.cc:1398] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.673938 8631 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:34.674049 8631 ts_tablet_manager.cc:1367] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:34.674510 8631 raft_consensus.cc:348] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.674631 8631 raft_consensus.cc:374] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.674666 8631 raft_consensus.cc:729] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.674788 8631 consensus_queue.cc:260] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.675055 8631 ts_tablet_manager.cc:1398] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.688225 8632 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:34.689092 8634 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:34.689209 8632 tablet_bootstrap.cc:654] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.689931 8634 tablet_bootstrap.cc:654] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.690368 8632 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:34.690464 8632 ts_tablet_manager.cc:1367] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:34.690889 8632 raft_consensus.cc:348] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.690970 8634 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:34.691013 8632 raft_consensus.cc:374] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.691047 8632 raft_consensus.cc:729] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.691046 8634 ts_tablet_manager.cc:1367] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:34.691174 8632 consensus_queue.cc:260] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.691473 8632 ts_tablet_manager.cc:1398] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.691423 8634 raft_consensus.cc:348] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.691529 8634 raft_consensus.cc:374] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.691558 8634 raft_consensus.cc:729] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.691638 8632 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:34.691670 8634 consensus_queue.cc:260] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.691933 8634 ts_tablet_manager.cc:1398] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.692101 8634 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:34.692422 8632 tablet_bootstrap.cc:654] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.692909 8634 tablet_bootstrap.cc:654] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.693432 8632 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:34.693506 8632 ts_tablet_manager.cc:1367] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:34.693883 8632 raft_consensus.cc:348] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.693998 8632 raft_consensus.cc:374] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.694038 8632 raft_consensus.cc:729] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.694161 8632 consensus_queue.cc:260] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.694420 8632 ts_tablet_manager.cc:1398] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.694567 8632 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:34.695101 8634 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:34.695190 8634 ts_tablet_manager.cc:1367] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:34.695341 8632 tablet_bootstrap.cc:654] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.695529 8634 raft_consensus.cc:348] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.695612 8634 raft_consensus.cc:374] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.695632 8634 raft_consensus.cc:729] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.695744 8634 consensus_queue.cc:260] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.696013 8634 ts_tablet_manager.cc:1398] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.696238 8634 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:34.696358 8632 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:34.696442 8632 ts_tablet_manager.cc:1367] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:34.696858 8632 raft_consensus.cc:348] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.696955 8632 raft_consensus.cc:374] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.696985 8632 raft_consensus.cc:729] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.697091 8632 consensus_queue.cc:260] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.697361 8632 ts_tablet_manager.cc:1398] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.698179 8634 tablet_bootstrap.cc:654] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.699595 8634 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:34.699684 8634 ts_tablet_manager.cc:1367] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:34.700043 8634 raft_consensus.cc:348] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.700141 8634 raft_consensus.cc:374] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.700168 8634 raft_consensus.cc:729] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.700291 8634 consensus_queue.cc:260] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.700552 8634 ts_tablet_manager.cc:1398] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.749027 8684 raft_consensus.cc:482] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.749210 8684 raft_consensus.cc:504] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.749696 8684 leader_election.cc:288] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:34.750188 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:34.750203 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:34.750352 7298 raft_consensus.cc:2454] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:34.750360 7431 raft_consensus.cc:2454] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:34.750685 7627 leader_election.cc:302] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:34.750900 8684 raft_consensus.cc:2781] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.750972 8684 raft_consensus.cc:482] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.751006 8684 raft_consensus.cc:3037] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.751986 8684 raft_consensus.cc:504] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.752393 8684 leader_election.cc:288] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:34.752799 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:34.752816 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:34.752943 7431 raft_consensus.cc:3037] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.752962 7298 raft_consensus.cc:3037] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.753798 7298 raft_consensus.cc:2454] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:34.753966 7431 raft_consensus.cc:2454] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:34.754496 7626 leader_election.cc:302] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:34.755026 8684 raft_consensus.cc:2781] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.755125 8684 raft_consensus.cc:686] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:34.755276 8684 consensus_queue.cc:237] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:34.757232 7157 catalog_manager.cc:5484] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:34.887257 8681 raft_consensus.cc:482] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.887406 8681 raft_consensus.cc:504] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.887929 8681 leader_election.cc:288] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:34.887964 8684 raft_consensus.cc:482] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.888217 8684 raft_consensus.cc:504] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.888420 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86634d81241c44cebcb602a6982f4244" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:34.888446 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86634d81241c44cebcb602a6982f4244" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:34.888553 7693 raft_consensus.cc:2454] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:34.888564 7298 raft_consensus.cc:2454] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:34.888864 7495 leader_election.cc:302] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:34.888900 8684 leader_election.cc:288] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:34.889048 8681 raft_consensus.cc:2781] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.889101 8681 raft_consensus.cc:482] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.889148 8681 raft_consensus.cc:3037] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.889194 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4582c127b7d34e08a68ea9094de11f19" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:34.889338 7431 raft_consensus.cc:2454] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:34.889434 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4582c127b7d34e08a68ea9094de11f19" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:34.889564 7298 raft_consensus.cc:2454] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:34.889613 7627 leader_election.cc:302] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:34.889760 8684 raft_consensus.cc:2781] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.889809 8684 raft_consensus.cc:482] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.889844 8684 raft_consensus.cc:3037] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.890293 8681 raft_consensus.cc:504] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.890606 8681 leader_election.cc:288] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:34.890628 8684 raft_consensus.cc:504] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.890918 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86634d81241c44cebcb602a6982f4244" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:34.890960 8684 leader_election.cc:288] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:34.890939 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86634d81241c44cebcb602a6982f4244" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:34.891047 7298 raft_consensus.cc:3037] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.891060 7693 raft_consensus.cc:3037] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.891407 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4582c127b7d34e08a68ea9094de11f19" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:34.891525 7299 raft_consensus.cc:3037] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.892004 7693 raft_consensus.cc:2454] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:34.892015 7298 raft_consensus.cc:2454] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:34.892253 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4582c127b7d34e08a68ea9094de11f19" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:34.892298 7495 leader_election.cc:302] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:34.892385 7431 raft_consensus.cc:3037] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.892400 7299 raft_consensus.cc:2454] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:34.892613 8681 raft_consensus.cc:2781] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.892668 7626 leader_election.cc:302] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:34.892697 8681 raft_consensus.cc:686] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:34.892812 8684 raft_consensus.cc:2781] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.892822 8681 consensus_queue.cc:237] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:34.892930 8684 raft_consensus.cc:686] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:34.893055 8684 consensus_queue.cc:237] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:34.893360 7431 raft_consensus.cc:2454] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:34.894699 7157 catalog_manager.cc:5484] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:34.894955 7156 catalog_manager.cc:5484] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:35.018206 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:35.018499 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:35.021787 8698 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 13:59:35.108609 7693 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.109390 7431 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.109795 7299 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.109826 8643 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.110452 8571 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.110776 8571 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.117460 7299 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.117902 8679 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.126545 8437 mvcc.cc:205] Tried to move back new op lower bound from 7023594598858838016 to 7023594597962473472. Current Snapshot: MvccSnapshot[applied={T|T < 7023594598858838016 or (T in {7023594598858838016})}]
I20240503 13:59:35.137612 8737 mvcc.cc:205] Tried to move back new op lower bound from 7023594598844530688 to 7023594597963141120. Current Snapshot: MvccSnapshot[applied={T|T < 7023594598844530688}]
I20240503 13:59:35.149426 7298 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.150580 8684 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.152034 7431 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.152683 8684 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.160276 8613 mvcc.cc:205] Tried to move back new op lower bound from 7023594599007997952 to 7023594597399154688. Current Snapshot: MvccSnapshot[applied={T|T < 7023594599007997952}]
INSERT report
rows total: 2000
time total: 116.926 ms
time per row: 0.0584632 ms
I20240503 13:59:35.184129 7153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:35.189721 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 78bb6d55e8b44732b32da55f00071b21 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.190093 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bb6d55e8b44732b32da55f00071b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.190045 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 93d454886dfa40018a4403a3d74b3de2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.190277 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d454886dfa40018a4403a3d74b3de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.190299 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 78bb6d55e8b44732b32da55f00071b21 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.190585 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bb6d55e8b44732b32da55f00071b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.190774 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 93d454886dfa40018a4403a3d74b3de2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.191021 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d454886dfa40018a4403a3d74b3de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.192703 7542 tablet_service.cc:1467] Processing CreateTablet for tablet c1a85f321bfa4faf9b48cb319221fb94 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.192937 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a85f321bfa4faf9b48cb319221fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.193210 8632 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:35.193806 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 93d454886dfa40018a4403a3d74b3de2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.193825 7672 tablet_service.cc:1467] Processing CreateTablet for tablet c1a85f321bfa4faf9b48cb319221fb94 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.194149 8632 tablet_bootstrap.cc:654] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.189934 7279 tablet_service.cc:1467] Processing CreateTablet for tablet 78bb6d55e8b44732b32da55f00071b21 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.194530 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bb6d55e8b44732b32da55f00071b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.195457 8632 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:35.195482 8743 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:35.195557 8632 ts_tablet_manager.cc:1367] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.196019 8632 raft_consensus.cc:348] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.196131 8632 raft_consensus.cc:374] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.196163 8632 raft_consensus.cc:729] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.196291 8632 consensus_queue.cc:260] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.196403 8743 tablet_bootstrap.cc:654] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.196586 8632 ts_tablet_manager.cc:1398] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.196765 8632 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:35.194139 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d454886dfa40018a4403a3d74b3de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.197194 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a85f321bfa4faf9b48cb319221fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.197597 8632 tablet_bootstrap.cc:654] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.198761 8632 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:35.198832 8632 ts_tablet_manager.cc:1367] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.199258 8632 raft_consensus.cc:348] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.199352 8632 raft_consensus.cc:374] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.199381 8632 raft_consensus.cc:729] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.199492 8634 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:35.199503 8632 consensus_queue.cc:260] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.199780 8632 ts_tablet_manager.cc:1398] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.199760 7410 tablet_service.cc:1467] Processing CreateTablet for tablet c1a85f321bfa4faf9b48cb319221fb94 (DEFAULT_TABLE table=rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.200001 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a85f321bfa4faf9b48cb319221fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.200212 8634 tablet_bootstrap.cc:654] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.201807 8634 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:35.201891 8634 ts_tablet_manager.cc:1367] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.202383 8634 raft_consensus.cc:348] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.202503 8634 raft_consensus.cc:374] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.202534 8634 raft_consensus.cc:729] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.202565 8742 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:35.202667 8634 consensus_queue.cc:260] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.202942 8634 ts_tablet_manager.cc:1398] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.203130 8634 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:35.203517 8742 tablet_bootstrap.cc:654] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.203965 8634 tablet_bootstrap.cc:654] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.204528 8743 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:35.204625 8743 ts_tablet_manager.cc:1367] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:59:35.205102 8743 raft_consensus.cc:348] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.205206 8743 raft_consensus.cc:374] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.205192 8634 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:35.205226 8743 raft_consensus.cc:729] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.205266 8634 ts_tablet_manager.cc:1367] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:35.205327 8743 consensus_queue.cc:260] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.205590 8743 ts_tablet_manager.cc:1398] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.205741 8743 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:35.205701 8634 raft_consensus.cc:348] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.205798 8634 raft_consensus.cc:374] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.205832 8634 raft_consensus.cc:729] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.205933 8634 consensus_queue.cc:260] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.206311 8634 ts_tablet_manager.cc:1398] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.206696 8743 tablet_bootstrap.cc:654] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.208156 8742 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:35.208271 8742 ts_tablet_manager.cc:1367] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:59:35.208772 8742 raft_consensus.cc:348] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.208897 8742 raft_consensus.cc:374] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.208930 8742 raft_consensus.cc:729] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.209053 8742 consensus_queue.cc:260] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.209345 8742 ts_tablet_manager.cc:1398] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.209373 8743 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:35.209445 8743 ts_tablet_manager.cc:1367] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:35.209539 8742 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:35.209897 8743 raft_consensus.cc:348] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.210032 8743 raft_consensus.cc:374] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.210063 8743 raft_consensus.cc:729] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.210187 8743 consensus_queue.cc:260] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.210456 8743 ts_tablet_manager.cc:1398] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.210489 8742 tablet_bootstrap.cc:654] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.211895 8742 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:35.211982 8742 ts_tablet_manager.cc:1367] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:35.212375 8742 raft_consensus.cc:348] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.212457 8742 raft_consensus.cc:374] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.212476 8742 raft_consensus.cc:729] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.212572 8742 consensus_queue.cc:260] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.212833 8742 ts_tablet_manager.cc:1398] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.213006 8742 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:35.213934 8742 tablet_bootstrap.cc:654] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.214951 8742 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:35.215082 8742 ts_tablet_manager.cc:1367] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:35.215523 8742 raft_consensus.cc:348] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.215610 8742 raft_consensus.cc:374] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.215631 8742 raft_consensus.cc:729] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.215718 8742 consensus_queue.cc:260] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.215950 8742 ts_tablet_manager.cc:1398] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.237704 8357 raft_consensus.cc:482] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.237864 8357 raft_consensus.cc:504] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.238365 8357 leader_election.cc:288] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:35.238770 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a85f321bfa4faf9b48cb319221fb94" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:35.238770 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a85f321bfa4faf9b48cb319221fb94" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:35.238924 7562 raft_consensus.cc:2454] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:35.238924 7693 raft_consensus.cc:2454] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 0.
I20240503 13:59:35.239212 7364 leader_election.cc:302] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:35.239387 8357 raft_consensus.cc:2781] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.239445 8357 raft_consensus.cc:482] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.239473 8357 raft_consensus.cc:3037] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.240309 8357 raft_consensus.cc:504] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.240658 8357 leader_election.cc:288] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:35.241026 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a85f321bfa4faf9b48cb319221fb94" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:35.241026 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a85f321bfa4faf9b48cb319221fb94" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:35.241164 7562 raft_consensus.cc:3037] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.241194 7693 raft_consensus.cc:3037] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.242161 7693 raft_consensus.cc:2454] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:35.242161 7562 raft_consensus.cc:2454] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 1.
I20240503 13:59:35.242467 7364 leader_election.cc:302] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:35.242648 8357 raft_consensus.cc:2781] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.242820 8357 raft_consensus.cc:686] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:35.242959 8357 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:35.244786 7153 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.452433 8729 raft_consensus.cc:482] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.452587 8729 raft_consensus.cc:504] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.453033 8729 leader_election.cc:288] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:35.453532 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d454886dfa40018a4403a3d74b3de2" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:35.453532 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d454886dfa40018a4403a3d74b3de2" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:35.453696 7562 raft_consensus.cc:2454] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:35.453697 7298 raft_consensus.cc:2454] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:35.454036 7626 leader_election.cc:302] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:35.454234 8729 raft_consensus.cc:2781] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.454298 8729 raft_consensus.cc:482] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.454341 8729 raft_consensus.cc:3037] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.455206 8729 raft_consensus.cc:504] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.455538 8729 leader_election.cc:288] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:35.455916 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d454886dfa40018a4403a3d74b3de2" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:35.456032 7562 raft_consensus.cc:3037] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.455971 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d454886dfa40018a4403a3d74b3de2" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:35.456149 7298 raft_consensus.cc:3037] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.456813 7562 raft_consensus.cc:2454] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:35.456936 7298 raft_consensus.cc:2454] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:35.457163 7626 leader_election.cc:302] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:35.457321 8729 raft_consensus.cc:2781] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.457403 8729 raft_consensus.cc:686] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:35.457540 8729 consensus_queue.cc:237] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.459323 7153 catalog_manager.cc:5484] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.501688 8735 raft_consensus.cc:482] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.501842 8735 raft_consensus.cc:504] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.502321 8735 leader_election.cc:288] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:35.502835 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bb6d55e8b44732b32da55f00071b21" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:35.502856 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bb6d55e8b44732b32da55f00071b21" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:35.502972 7562 raft_consensus.cc:2454] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:35.503018 7431 raft_consensus.cc:2454] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:35.503288 7233 leader_election.cc:302] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.503482 8735 raft_consensus.cc:2781] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.503551 8735 raft_consensus.cc:482] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.503607 8735 raft_consensus.cc:3037] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.504674 8735 raft_consensus.cc:504] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.505151 8735 leader_election.cc:288] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:35.505481 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bb6d55e8b44732b32da55f00071b21" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:35.505594 7431 raft_consensus.cc:3037] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.505549 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bb6d55e8b44732b32da55f00071b21" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:35.505678 7562 raft_consensus.cc:3037] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.506376 7431 raft_consensus.cc:2454] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:35.506513 7562 raft_consensus.cc:2454] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:35.506646 7234 leader_election.cc:302] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.506812 8735 raft_consensus.cc:2781] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.506892 8735 raft_consensus.cc:686] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:35.507022 8735 consensus_queue.cc:237] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.508813 7153 catalog_manager.cc:5484] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:35.675465 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:35.675747 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:35.679001 8757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 13:59:35.700897 8357 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.705843 8753 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.777942 7562 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.778843 7298 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.779372 8729 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.779721 8729 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.781499 7431 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.781960 8611 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.792474 7561 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.792907 8611 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.767 ms
time per row: 0.0613836 ms
I20240503 13:59:35.849630 7149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:35.855479 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 491ee8399ef04ae7b93e3228fef05eb0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.855683 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 14304180134241c792ec1f23ff89f94c (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.855850 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ee8399ef04ae7b93e3228fef05eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.855933 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14304180134241c792ec1f23ff89f94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.856060 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 23cf5703ccef47ea927342b9c29f820a (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.856374 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23cf5703ccef47ea927342b9c29f820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.856666 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 14304180134241c792ec1f23ff89f94c (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.856922 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14304180134241c792ec1f23ff89f94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.858428 7279 tablet_service.cc:1467] Processing CreateTablet for tablet 491ee8399ef04ae7b93e3228fef05eb0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.858444 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 23cf5703ccef47ea927342b9c29f820a (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.856060 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 14304180134241c792ec1f23ff89f94c (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.858668 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ee8399ef04ae7b93e3228fef05eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.858667 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23cf5703ccef47ea927342b9c29f820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.858773 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14304180134241c792ec1f23ff89f94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.860760 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 491ee8399ef04ae7b93e3228fef05eb0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.860843 8820 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:35.860833 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 23cf5703ccef47ea927342b9c29f820a (DEFAULT_TABLE table=rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.861014 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ee8399ef04ae7b93e3228fef05eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.861064 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23cf5703ccef47ea927342b9c29f820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.861085 8819 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:35.861728 8820 tablet_bootstrap.cc:654] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.861889 8819 tablet_bootstrap.cc:654] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.862726 8821 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:35.863664 8821 tablet_bootstrap.cc:654] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.863803 8818 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:35.864043 8820 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:35.864133 8820 ts_tablet_manager.cc:1367] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.864184 8819 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:35.864257 8819 ts_tablet_manager.cc:1367] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.864588 8820 raft_consensus.cc:348] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.864671 8818 tablet_bootstrap.cc:654] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.864701 8820 raft_consensus.cc:374] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.864661 8819 raft_consensus.cc:348] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.864730 8820 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.864780 8819 raft_consensus.cc:374] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.864810 8819 raft_consensus.cc:729] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.864861 8820 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.864915 8819 consensus_queue.cc:260] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.865146 8820 ts_tablet_manager.cc:1398] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.865259 8819 ts_tablet_manager.cc:1398] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.865290 8820 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:35.865399 8819 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:35.866170 8820 tablet_bootstrap.cc:654] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.866174 8819 tablet_bootstrap.cc:654] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.866770 8818 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:35.866863 8818 ts_tablet_manager.cc:1367] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.867983 8821 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:35.868077 8821 ts_tablet_manager.cc:1367] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:35.868086 8820 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:35.868160 8820 ts_tablet_manager.cc:1367] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:35.868553 8821 raft_consensus.cc:348] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.868680 8821 raft_consensus.cc:374] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.868718 8821 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.868830 8821 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.868860 8818 raft_consensus.cc:348] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.868973 8818 raft_consensus.cc:374] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.869001 8818 raft_consensus.cc:729] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.869110 8821 ts_tablet_manager.cc:1398] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.869118 8818 consensus_queue.cc:260] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.869266 8821 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:35.869231 8820 raft_consensus.cc:348] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.869318 8820 raft_consensus.cc:374] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.869347 8820 raft_consensus.cc:729] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.869380 8818 ts_tablet_manager.cc:1398] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:59:35.869450 8820 consensus_queue.cc:260] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.869553 8818 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:35.869793 8820 ts_tablet_manager.cc:1398] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.869944 8819 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:35.870060 8819 ts_tablet_manager.cc:1367] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:35.870167 8821 tablet_bootstrap.cc:654] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.870422 8818 tablet_bootstrap.cc:654] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.870481 8819 raft_consensus.cc:348] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.870585 8819 raft_consensus.cc:374] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.870616 8819 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.870743 8819 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.871098 8819 ts_tablet_manager.cc:1398] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.872823 8818 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:35.872998 8818 ts_tablet_manager.cc:1367] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.873510 8818 raft_consensus.cc:348] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.872823 8821 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:35.873737 8821 ts_tablet_manager.cc:1367] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:35.872931 8805 raft_consensus.cc:482] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.874223 8821 raft_consensus.cc:348] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.874258 8805 raft_consensus.cc:504] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.874327 8821 raft_consensus.cc:374] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.874361 8821 raft_consensus.cc:729] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.874477 8821 consensus_queue.cc:260] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.874644 8805 leader_election.cc:288] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:35.874748 8821 ts_tablet_manager.cc:1398] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.875025 8821 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:35.875082 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14304180134241c792ec1f23ff89f94c" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:35.875144 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14304180134241c792ec1f23ff89f94c" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:35.875205 7693 raft_consensus.cc:2454] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:35.875262 7431 raft_consensus.cc:2454] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:35.875491 7234 leader_election.cc:302] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.875675 8611 raft_consensus.cc:2781] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.875731 8611 raft_consensus.cc:482] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.875767 8611 raft_consensus.cc:3037] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.876042 8818 raft_consensus.cc:374] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.876107 8818 raft_consensus.cc:729] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.876243 8821 tablet_bootstrap.cc:654] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.876257 8818 consensus_queue.cc:260] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.876787 8611 raft_consensus.cc:504] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.876991 8818 ts_tablet_manager.cc:1398] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:35.877137 8611 leader_election.cc:288] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:35.877470 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14304180134241c792ec1f23ff89f94c" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:35.877512 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14304180134241c792ec1f23ff89f94c" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:35.877599 7693 raft_consensus.cc:3037] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.877626 7431 raft_consensus.cc:3037] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.878278 8821 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:35.878374 8821 ts_tablet_manager.cc:1367] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:35.878618 7693 raft_consensus.cc:2454] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:35.878656 7431 raft_consensus.cc:2454] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:35.878816 8821 raft_consensus.cc:348] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.878909 8821 raft_consensus.cc:374] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.878931 7234 leader_election.cc:302] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.879104 8611 raft_consensus.cc:2781] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.879176 8611 raft_consensus.cc:686] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:35.879297 8611 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.878937 8821 raft_consensus.cc:729] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.879732 8821 consensus_queue.cc:260] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.880116 8821 ts_tablet_manager.cc:1398] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.880896 7153 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.988394 8556 raft_consensus.cc:482] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.988529 8556 raft_consensus.cc:504] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.988898 8556 leader_election.cc:288] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:35.989290 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ee8399ef04ae7b93e3228fef05eb0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:35.989316 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ee8399ef04ae7b93e3228fef05eb0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:35.989439 7298 raft_consensus.cc:2454] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:35.989464 7431 raft_consensus.cc:2454] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:35.989760 7496 leader_election.cc:302] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:35.989923 8556 raft_consensus.cc:2781] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.990015 8556 raft_consensus.cc:482] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.990064 8556 raft_consensus.cc:3037] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.990926 8611 raft_consensus.cc:482] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.990942 8556 raft_consensus.cc:504] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.991015 8611 raft_consensus.cc:504] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.991312 8556 leader_election.cc:288] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:35.991391 8611 leader_election.cc:288] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:35.991632 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ee8399ef04ae7b93e3228fef05eb0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:35.991760 7298 raft_consensus.cc:3037] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.991770 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491ee8399ef04ae7b93e3228fef05eb0" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:35.991779 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:35.991883 7431 raft_consensus.cc:3037] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.991894 7561 raft_consensus.cc:2454] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:35.992802 7431 raft_consensus.cc:2454] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:35.992801 7298 raft_consensus.cc:2454] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:35.993026 7233 leader_election.cc:302] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.993099 7495 leader_election.cc:302] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.993201 8611 raft_consensus.cc:2781] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.993255 8611 raft_consensus.cc:482] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.993268 8556 raft_consensus.cc:2781] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.993294 8611 raft_consensus.cc:3037] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.993340 8556 raft_consensus.cc:686] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:35.993312 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:35.993444 7693 raft_consensus.cc:2454] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 0.
I20240503 13:59:35.993487 8556 consensus_queue.cc:237] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } }
I20240503 13:59:35.994226 8611 raft_consensus.cc:504] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.994571 8611 leader_election.cc:288] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:35.994946 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:35.995055 7561 raft_consensus.cc:3037] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.995139 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:35.995272 7693 raft_consensus.cc:3037] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.995857 7561 raft_consensus.cc:2454] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:35.995884 7153 catalog_manager.cc:5484] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:35.996114 7233 leader_election.cc:302] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:35.996268 8611 raft_consensus.cc:2781] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.996284 7693 raft_consensus.cc:2454] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 1.
I20240503 13:59:35.996353 8611 raft_consensus.cc:686] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Leader. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:35.996485 8611 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }
I20240503 13:59:35.998414 7149 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:36.127096 8834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:36.127367 8834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:36.130823 8834 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 13:59:36.211429 7431 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.211683 7693 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.212271 7298 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.212404 8611 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.212707 8681 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.213220 7561 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.213560 8735 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.214917 7692 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.215198 8805 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.216656 7431 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.217165 8681 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.219429 8735 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.981 ms
time per row: 0.0504904 ms
I20240503 13:59:36.278149 7155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60138:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:36.284304 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 63552cb8c9c94a07b2a0b157402efa7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.284415 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 63552cb8c9c94a07b2a0b157402efa7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.284657 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63552cb8c9c94a07b2a0b157402efa7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.284657 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63552cb8c9c94a07b2a0b157402efa7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.285053 7672 tablet_service.cc:1467] Processing CreateTablet for tablet e0b7095bdf3549fb83b3573a8e65deee (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.285312 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b7095bdf3549fb83b3573a8e65deee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.285248 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 110ac05b2e8c4594a03225ab2c1ad56d (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.285513 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110ac05b2e8c4594a03225ab2c1ad56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.286949 8820 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:36.287181 8819 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:36.287681 8821 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:36.287894 8820 tablet_bootstrap.cc:654] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.287955 8819 tablet_bootstrap.cc:654] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.288534 8821 tablet_bootstrap.cc:654] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.288651 7410 tablet_service.cc:1467] Processing CreateTablet for tablet e0b7095bdf3549fb83b3573a8e65deee (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.288872 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b7095bdf3549fb83b3573a8e65deee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.288941 8820 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:36.289008 8820 ts_tablet_manager.cc:1367] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.289096 7541 tablet_service.cc:1467] Processing CreateTablet for tablet e0b7095bdf3549fb83b3573a8e65deee (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.289367 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b7095bdf3549fb83b3573a8e65deee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.289448 8820 raft_consensus.cc:348] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.289499 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 63552cb8c9c94a07b2a0b157402efa7a (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.289551 8820 raft_consensus.cc:374] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.289580 8820 raft_consensus.cc:729] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.289698 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63552cb8c9c94a07b2a0b157402efa7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.289717 8820 consensus_queue.cc:260] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.290022 8820 ts_tablet_manager.cc:1398] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.290249 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 110ac05b2e8c4594a03225ab2c1ad56d (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.290275 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 110ac05b2e8c4594a03225ab2c1ad56d (DEFAULT_TABLE table=rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.290469 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110ac05b2e8c4594a03225ab2c1ad56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.290479 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110ac05b2e8c4594a03225ab2c1ad56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.291509 8818 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:36.292316 8820 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:36.292438 8818 tablet_bootstrap.cc:654] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.293144 8820 tablet_bootstrap.cc:654] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.294129 8821 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96: No bootstrap required, opened a new log
I20240503 13:59:36.294237 8821 ts_tablet_manager.cc:1367] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:36.294642 8820 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:36.294713 8820 ts_tablet_manager.cc:1367] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.294745 8821 raft_consensus.cc:348] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.294859 8821 raft_consensus.cc:374] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.294903 8821 raft_consensus.cc:729] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.295029 8821 consensus_queue.cc:260] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.295157 8820 raft_consensus.cc:348] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.295243 8820 raft_consensus.cc:374] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.295272 8820 raft_consensus.cc:729] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.295408 8819 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:36.295408 8820 consensus_queue.cc:260] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.295511 8819 ts_tablet_manager.cc:1367] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 13:59:36.295805 8821 ts_tablet_manager.cc:1398] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.295960 8819 raft_consensus.cc:348] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.296082 8819 raft_consensus.cc:374] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.296115 8819 raft_consensus.cc:729] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.296159 8820 ts_tablet_manager.cc:1398] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.296170 8818 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:36.296260 8818 ts_tablet_manager.cc:1367] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:36.296239 8819 consensus_queue.cc:260] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.296326 8820 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:36.296519 8819 ts_tablet_manager.cc:1398] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.296705 8819 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:36.296733 8818 raft_consensus.cc:348] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.296833 8818 raft_consensus.cc:374] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.296867 8818 raft_consensus.cc:729] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.296986 8818 consensus_queue.cc:260] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.297272 8818 ts_tablet_manager.cc:1398] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.297355 8820 tablet_bootstrap.cc:654] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.297578 8819 tablet_bootstrap.cc:654] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.298732 8819 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: No bootstrap required, opened a new log
I20240503 13:59:36.298818 8819 ts_tablet_manager.cc:1367] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.299229 8819 raft_consensus.cc:348] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.299331 8819 raft_consensus.cc:374] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.299360 8819 raft_consensus.cc:729] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.299469 8819 consensus_queue.cc:260] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.299753 8819 ts_tablet_manager.cc:1398] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.300027 8818 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:36.301797 8818 tablet_bootstrap.cc:654] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.303022 8818 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:36.303117 8818 ts_tablet_manager.cc:1367] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:36.303630 8818 raft_consensus.cc:348] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.303766 8818 raft_consensus.cc:374] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.303797 8818 raft_consensus.cc:729] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.303913 8818 consensus_queue.cc:260] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.304203 8818 ts_tablet_manager.cc:1398] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.304373 8818 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:36.304443 8820 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: No bootstrap required, opened a new log
I20240503 13:59:36.304541 8820 ts_tablet_manager.cc:1367] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:36.305016 8820 raft_consensus.cc:348] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.305145 8820 raft_consensus.cc:374] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.305177 8820 raft_consensus.cc:729] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.305253 8818 tablet_bootstrap.cc:654] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.305303 8820 consensus_queue.cc:260] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.305627 8820 ts_tablet_manager.cc:1398] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.306955 8818 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: No bootstrap required, opened a new log
I20240503 13:59:36.307061 8818 ts_tablet_manager.cc:1367] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:36.307547 8818 raft_consensus.cc:348] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.307657 8818 raft_consensus.cc:374] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.307688 8818 raft_consensus.cc:729] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.307845 8818 consensus_queue.cc:260] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.308154 8818 ts_tablet_manager.cc:1398] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:36.388774 8556 raft_consensus.cc:482] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.388931 8556 raft_consensus.cc:504] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.389384 8556 leader_election.cc:288] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:36.389863 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" is_pre_election: true
I20240503 13:59:36.389885 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:36.390055 7299 raft_consensus.cc:2454] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:36.390072 7429 raft_consensus.cc:2454] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:36.390405 7496 leader_election.cc:302] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:36.390630 8556 raft_consensus.cc:2781] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.390715 8556 raft_consensus.cc:482] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.390751 8556 raft_consensus.cc:3037] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.391794 8556 raft_consensus.cc:504] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.392237 8556 leader_election.cc:288] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:36.392608 7299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:36.392607 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:36.392729 7429 raft_consensus.cc:3037] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.392728 7299 raft_consensus.cc:3037] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.393563 7299 raft_consensus.cc:2454] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:36.393740 7429 raft_consensus.cc:2454] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:36.394080 7496 leader_election.cc:302] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:36.394335 8556 raft_consensus.cc:2781] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.394423 8556 raft_consensus.cc:686] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:36.394714 8556 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.397027 7155 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:36.493012 8571 raft_consensus.cc:482] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.493168 8571 raft_consensus.cc:504] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.493614 8571 leader_election.cc:288] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:36.494100 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b7095bdf3549fb83b3573a8e65deee" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:36.494102 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b7095bdf3549fb83b3573a8e65deee" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2" is_pre_election: true
I20240503 13:59:36.494249 7561 raft_consensus.cc:2454] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:36.494251 7429 raft_consensus.cc:2454] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 0.
I20240503 13:59:36.494537 7627 leader_election.cc:302] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:36.494735 8571 raft_consensus.cc:2781] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.494802 8571 raft_consensus.cc:482] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.494837 8571 raft_consensus.cc:3037] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.495710 8571 raft_consensus.cc:504] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.496073 8571 leader_election.cc:288] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Requested vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:36.496465 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b7095bdf3549fb83b3573a8e65deee" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:36.496465 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b7095bdf3549fb83b3573a8e65deee" candidate_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:36.496619 7561 raft_consensus.cc:3037] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.496634 7429 raft_consensus.cc:3037] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.497642 7561 raft_consensus.cc:2454] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:36.497638 7429 raft_consensus.cc:2454] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6f98c1ec0346ebb30c59ffe8d7c29a in term 1.
I20240503 13:59:36.497960 7626 leader_election.cc:302] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6f98c1ec0346ebb30c59ffe8d7c29a, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:36.498173 8571 raft_consensus.cc:2781] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.498260 8571 raft_consensus.cc:686] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:36.498404 8571 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } }
I20240503 13:59:36.500303 7155 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:36.508919 8556 raft_consensus.cc:482] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.509054 8556 raft_consensus.cc:504] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.509478 8556 leader_election.cc:288] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:36.510013 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63552cb8c9c94a07b2a0b157402efa7a" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576" is_pre_election: true
I20240503 13:59:36.510037 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63552cb8c9c94a07b2a0b157402efa7a" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" is_pre_election: true
I20240503 13:59:36.510179 7693 raft_consensus.cc:2454] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:36.510174 7429 raft_consensus.cc:2454] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 0.
I20240503 13:59:36.510525 7496 leader_election.cc:302] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:36.510730 8556 raft_consensus.cc:2781] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.510794 8556 raft_consensus.cc:482] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.510838 8556 raft_consensus.cc:3037] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.511638 8556 raft_consensus.cc:504] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.511991 8556 leader_election.cc:288] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Requested vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:36.512406 7429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63552cb8c9c94a07b2a0b157402efa7a" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:36.512444 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63552cb8c9c94a07b2a0b157402efa7a" candidate_uuid: "9d921974718b4613b136eee9d0e1bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:36.512547 7693 raft_consensus.cc:3037] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.512557 7429 raft_consensus.cc:3037] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.513320 7693 raft_consensus.cc:2454] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:36.513360 7429 raft_consensus.cc:2454] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d921974718b4613b136eee9d0e1bad2 in term 1.
I20240503 13:59:36.513614 7496 leader_election.cc:302] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2; no voters:
I20240503 13:59:36.513789 8556 raft_consensus.cc:2781] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.513876 8556 raft_consensus.cc:686] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:36.514055 8556 consensus_queue.cc:237] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } }
I20240503 13:59:36.515923 7155 catalog_manager.cc:5484] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:36.659317 8884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:36.659642 8884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:36.662946 8884 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 13:59:36.763419 7561 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.763906 8571 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.764441 7429 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.764856 8571 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.766781 7693 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.767095 7430 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.767699 8681 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.768046 8681 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.773276 8556 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.780503 8681 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 135.565 ms
time per row: 0.0677826 ms
I20240503 13:59:36.842818 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:36553
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:45103
--webserver_port=42729
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:36.961222 8943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:36.961503 8943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:36.961572 8943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:36.965224 8943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:36.965319 8943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:36.965417 8943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:36.969655 8943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:36553
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=42729
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.8943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:36.970759 8943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:36.972210 8943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:36.981555 8948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:36.982043 8951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:36.982877 8949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:36.983539 8943 server_base.cc:1034] running on GCE node
I20240503 13:59:36.984019 8943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:36.984645 8943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:36.985821 8943 hybrid_clock.cc:648] HybridClock initialized: now 1714744776985796 us; error 43 us; skew 500 ppm
I20240503 13:59:36.988009 8943 webserver.cc:460] Webserver started at http://127.0.99.133:42729/ using document root <none> and password file <none>
I20240503 13:59:36.988618 8943 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:36.988695 8943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:36.992337 8943 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:36.994457 8957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:36.995599 8943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.995718 8943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "476f53c7e40c4a0b86ffbd818aea818d"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:36.996189 8943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.014470 8943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.015209 8943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.015377 8943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.015945 8943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.016919 8943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:37.016970 8943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.017025 8943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:37.017052 8943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.025643 8943 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36553
I20240503 13:59:37.025668 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36553 every 8 connection(s)
I20240503 13:59:37.026849 8943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:37.028744 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 8943
I20240503 13:59:37.029203 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:36383
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:45103
--webserver_port=34125
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:37.037257 9071 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:37.037569 9071 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.038277 9071 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:37.039510 7151 ts_manager.cc:194] Re-registered known tserver with Master: 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553)
I20240503 13:59:37.040108 7151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:56115
W20240503 13:59:37.147625 9074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.147917 9074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:37.147981 9074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.151376 9074 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:37.151465 9074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:37.151540 9074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:37.155473 9074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:36383
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=34125
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.9074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:37.156525 9074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:37.157872 9074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:37.167889 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.168660 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.169219 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:37.170154 9074 server_base.cc:1034] running on GCE node
I20240503 13:59:37.170639 9074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:37.171272 9074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:37.172437 9074 hybrid_clock.cc:648] HybridClock initialized: now 1714744777172419 us; error 38 us; skew 500 ppm
I20240503 13:59:37.174477 9074 webserver.cc:460] Webserver started at http://127.0.99.134:34125/ using document root <none> and password file <none>
I20240503 13:59:37.175056 9074 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:37.175124 9074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:37.178946 9074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:37.180932 9088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.182072 9074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:37.182191 9074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "708734e6729441fcb2c5952fa47771d9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:37.182646 9074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.196991 9074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.197625 9074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.197783 9074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.198396 9074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.199280 9074 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:37.199326 9074 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.199365 9074 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:37.199385 9074 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.207578 9074 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36383
I20240503 13:59:37.207597 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36383 every 8 connection(s)
I20240503 13:59:37.208699 9074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:37.215348 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 9074
I20240503 13:59:37.215790 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:41051
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:45103
--webserver_port=36633
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:37.218649 9202 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:37.218896 9202 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.219566 9202 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:37.220621 7151 ts_manager.cc:194] Re-registered known tserver with Master: 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383)
I20240503 13:59:37.221402 7151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:52897
W20240503 13:59:37.334832 9205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.335103 9205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:37.335155 9205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.338812 9205 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:37.338896 9205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:37.338985 9205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:37.343070 9205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:41051
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=36633
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.9205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:37.344105 9205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:37.345438 9205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:37.354991 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.355814 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.356323 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:37.357213 9205 server_base.cc:1034] running on GCE node
I20240503 13:59:37.357690 9205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:37.358379 9205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:37.359575 9205 hybrid_clock.cc:648] HybridClock initialized: now 1714744777359553 us; error 34 us; skew 500 ppm
I20240503 13:59:37.361860 9205 webserver.cc:460] Webserver started at http://127.0.99.135:36633/ using document root <none> and password file <none>
I20240503 13:59:37.362497 9205 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:37.362571 9205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:37.366214 9205 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 13:59:37.368294 9219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.369392 9205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.369532 9205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c5c3042421284f028fa512b7876da405"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:37.370143 9205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.399447 9205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.400203 9205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.400378 9205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.400976 9205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.401930 9205 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:37.402002 9205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.402065 9205 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:37.402087 9205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.411077 9205 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41051
I20240503 13:59:37.411099 9332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41051 every 8 connection(s)
I20240503 13:59:37.412235 9205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:37.412672 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 9205
I20240503 13:59:37.413129 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:43485
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:45103
--webserver_port=43353
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:37.423857 9333 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:37.424139 9333 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.424732 9333 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:37.425846 7151 ts_manager.cc:194] Re-registered known tserver with Master: c5c3042421284f028fa512b7876da405 (127.0.99.135:41051)
I20240503 13:59:37.426492 7151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:44003
W20240503 13:59:37.532389 9335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.532660 9335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:37.532714 9335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.536026 9335 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:37.536113 9335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:37.536207 9335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:59:37.540395 9335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:43485
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=43353
--tserver_master_addrs=127.0.99.190:45103
--never_fsync=true
--heap_profile_path=/tmp/kudu.9335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:37.541635 9335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:37.543161 9335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:37.552078 9343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.552843 9345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.553368 9342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:37.554236 9335 server_base.cc:1034] running on GCE node
I20240503 13:59:37.554641 9335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:37.555248 9335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:37.556413 9335 hybrid_clock.cc:648] HybridClock initialized: now 1714744777556385 us; error 46 us; skew 500 ppm
I20240503 13:59:37.558651 9335 webserver.cc:460] Webserver started at http://127.0.99.136:43353/ using document root <none> and password file <none>
I20240503 13:59:37.559242 9335 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:37.559312 9335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:37.562803 9335 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:37.565009 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.566303 9335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:37.566414 9335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "da390a1a407a4f30b217b3d60fba563f"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-6fwv"
I20240503 13:59:37.566874 9335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.579538 9335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.580179 9335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.580334 9335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.580891 9335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.581813 9335 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:37.581859 9335 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.581903 9335 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:37.581931 9335 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.590395 9335 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43485
I20240503 13:59:37.590433 9464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43485 every 8 connection(s)
I20240503 13:59:37.591499 9335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:59:37.600155 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 9335
I20240503 13:59:37.601315 9465 heartbeater.cc:347] Connected to a master server at 127.0.99.190:45103
I20240503 13:59:37.601594 9465 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.602802 9465 heartbeater.cc:511] Master 127.0.99.190:45103 requested a full tablet report, sending...
I20240503 13:59:37.604657 7151 ts_manager.cc:194] Re-registered known tserver with Master: da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485)
I20240503 13:59:37.605463 7151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:53829
I20240503 13:59:38.003031 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.004634 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.010663 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.022425 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.023017 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.027029 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.030200 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.042155 9071 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
I20240503 13:59:38.059144 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.078779 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.101126 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.101205 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.107645 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.131922 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.148591 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.152073 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.162488 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.163112 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.178401 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.183734 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.195957 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.206647 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.219187 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.221271 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.223404 9202 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
I20240503 13:59:38.227171 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.258464 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.284852 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.287117 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.296267 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.302656 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.309441 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.310160 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.314532 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.320489 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.339143 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.342392 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.350263 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.357966 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.359720 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.361727 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.363688 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.428835 9333 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
I20240503 13:59:38.580502 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.588987 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.610685 9465 heartbeater.cc:503] Master 127.0.99.190:45103 was elected leader, sending a full tablet report...
I20240503 13:59:38.615108 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.620290 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.627669 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.656577 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.659510 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.663245 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.665688 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.671474 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.698786 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.711014 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.722417 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.723332 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.731434 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.739430 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.751039 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.754482 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.762480 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.764782 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.771106 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.778333 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.788478 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.805907 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.809594 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.818373 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.819415 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.822098 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.839392 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.839949 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.844609 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.844712 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.849649 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.866684 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.877004 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.878326 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.890488 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.904841 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:38.905718 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:38.917930 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.099828 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.117530 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.120827 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.128767 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.135967 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.143594 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.152333 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.157568 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.159010 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.177456 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.186292 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.204574 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.207456 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.233569 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.235780 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.241585 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.246289 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.253087 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.273883 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.290850 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.298707 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.302429 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.309655 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.311779 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.313665 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.317836 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.322351 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.336102 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.354315 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.359582 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.380846 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.392423 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.431303 7561 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:39.433555 7431 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.434739 9972 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.435333 7299 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.436980 9972 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.440748 10197 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.443656 7431 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.445115 7299 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.448675 7156 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
W20240503 13:59:39.451136 7498 consensus_peers.cc:485] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aff688eaad64d149bdc1c8ab12f4ca1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.451763 7561 consensus_queue.cc:237] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.453745 7299 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.454175 7431 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.454459 9972 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.454684 10197 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.458287 9972 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
W20240503 13:59:39.460171 7498 consensus_peers.cc:485] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491ee8399ef04ae7b93e3228fef05eb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.460423 7299 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.480336 7561 consensus_queue.cc:237] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:39.480615 7431 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.482371 7693 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.482893 10197 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.487689 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.489027 7691 consensus_queue.cc:237] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:39.489280 7431 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:39.489598 7498 consensus_peers.cc:485] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63552cb8c9c94a07b2a0b157402efa7a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.489869 10201 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.490990 7693 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.492700 7562 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.493086 7299 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.493530 10082 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.493860 10082 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.497119 10201 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.497465 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.498422 7561 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.499001 8571 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.499897 7431 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.500443 10082 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.500844 7431 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.503647 10082 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.506999 7152 catalog_manager.cc:5484] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.507706 7691 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.507717 7429 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.511497 8571 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.512317 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.513258 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.513938 7299 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.515338 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.516458 7562 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.524142 7691 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.526571 7150 catalog_manager.cc:5484] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.526887 7150 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.527056 7150 catalog_manager.cc:5484] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.528923 7562 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.529500 8571 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.531523 7429 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.531955 10082 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.534142 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.539522 7561 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
W20240503 13:59:39.544720 7629 consensus_peers.cc:485] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b23f9eeeb4af45b7b679b70514e247b8. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:39.545091 7629 consensus_peers.cc:485] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0b7095bdf3549fb83b3573a8e65deee. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.547926 7299 consensus_queue.cc:237] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
W20240503 13:59:39.548543 7629 consensus_peers.cc:485] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9445bc10fdcf4e9b8d9167ba243bfda5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.549646 7561 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.550639 7431 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.551082 8805 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.552510 8805 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:39.554515 7236 consensus_peers.cc:485] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78bb6d55e8b44732b32da55f00071b21. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.554615 8805 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.555816 7561 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:39.556669 10246 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.557569 7299 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.557580 7429 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.557945 9670 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 13:59:39.559149 7498 consensus_peers.cc:485] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aff688eaad64d149bdc1c8ab12f4ca1. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:39.559248 7498 consensus_peers.cc:485] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aff688eaad64d149bdc1c8ab12f4ca1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.559320 9972 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.560782 9670 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.563551 7429 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.564354 7431 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.565660 7299 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.565872 7152 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.567250 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.567600 7156 catalog_manager.cc:5484] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.568166 7561 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.569821 7431 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.571969 7154 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.582785 7691 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:39.584959 7299 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.585047 7561 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.585419 10246 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.585698 10246 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 13:59:39.587054 7629 consensus_peers.cc:485] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b23f9eeeb4af45b7b679b70514e247b8. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:39.587153 7629 consensus_peers.cc:485] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b23f9eeeb4af45b7b679b70514e247b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.587522 10246 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.589913 7561 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.590682 7299 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.591202 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.592326 7154 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.603871 7431 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:39.605763 7299 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.606439 7691 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.606532 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.606989 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.609828 10299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.614192 7691 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
W20240503 13:59:39.615420 7367 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b7b43742e34431af0c16d320b35dd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.618522 7152 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.621044 7299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.628012 7562 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.635387 7691 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:39.637732 7562 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.637794 7431 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 13:59:39.638074 7629 consensus_peers.cc:485] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0b7095bdf3549fb83b3573a8e65deee. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.638235 10246 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.638789 10246 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.640623 10246 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
W20240503 13:59:39.642524 7629 consensus_peers.cc:485] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0b7095bdf3549fb83b3573a8e65deee. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.643769 7431 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.643996 7150 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.646658 7691 consensus_queue.cc:237] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.646864 7562 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.648269 7299 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.649041 10291 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:39.649488 7629 consensus_peers.cc:485] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4582c127b7d34e08a68ea9094de11f19. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.649783 7431 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.650293 8571 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.656983 7429 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:39.658687 7299 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.659839 8919 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.659740 10082 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.660885 7429 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.662480 7298 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.662925 7691 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.663105 7152 catalog_manager.cc:5484] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.663781 8919 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:39.666328 7367 consensus_peers.cc:485] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c74f828c28c64097ab4bf5ec66459d2f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.668243 7429 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.670437 7562 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.670902 8919 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.671106 10299 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.671545 7691 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.674096 7150 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
W20240503 13:59:39.674283 7367 consensus_peers.cc:485] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 902cf3db3581465c83170c4b13b3bffb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.674463 7298 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.675391 8919 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.679863 8919 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.682495 7298 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.683090 7152 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.687813 7562 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.689177 7429 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.691156 7299 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.691478 7691 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.691860 10298 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.692006 10299 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.692225 7299 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.694175 10298 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
W20240503 13:59:39.696478 7367 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b7b43742e34431af0c16d320b35dd7. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:39.696591 7367 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b7b43742e34431af0c16d320b35dd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.696496 7691 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.697227 7429 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:39.698058 7298 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.699092 7562 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.699483 8919 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:39.699656 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2212cf72b6ee4ee5b086455a75482e3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.699995 7150 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.703261 8919 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.705387 7562 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.706506 7691 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.707051 10298 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.708580 7691 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.709829 7152 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.714469 7298 consensus_queue.cc:237] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:39.716384 7562 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:39.716466 7236 consensus_peers.cc:485] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f73a5db0cae647d4b3375e6d42358307. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.716789 7429 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.716910 10142 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.717509 10142 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.719321 7298 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:39.721190 7691 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.721290 7429 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.726824 8805 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.727898 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.728323 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.728765 7298 consensus_queue.cc:237] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:39.729475 7562 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.731091 7561 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.732218 10276 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:39.732465 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14304180134241c792ec1f23ff89f94c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.733358 7154 catalog_manager.cc:5484] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:39.735632 7236 consensus_peers.cc:485] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3eaa7e5f641436a8f0cfdaf83443ca1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.735762 7691 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.736315 10276 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.736430 8805 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.737354 10276 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.738425 7691 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.739574 7561 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.743261 7693 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.743501 7431 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.743724 7150 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.748453 7156 catalog_manager.cc:5484] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.755009 7429 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.780170 7561 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:39.780334 7429 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
W20240503 13:59:39.782164 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade40ddaf0304b1a9e94db6630175c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.782516 7298 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.783672 7693 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.783687 9972 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.784037 7561 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.784446 10298 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.784699 10350 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.789233 7429 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.789659 9972 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:39.790264 7367 consensus_peers.cc:485] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fca185e6c04648c5aa4f4ad51b0e1a39. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.791779 10201 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.792297 7429 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.794624 7298 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.794688 7154 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.802012 10350 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.804505 7693 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.806298 7561 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.806401 7153 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.817404 7693 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
W20240503 13:59:39.819396 7629 consensus_peers.cc:485] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff246d9b2c594282bf2e99d2fd2b617d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.819875 7298 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.820314 10291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.822394 7561 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:39.823731 7429 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.824259 10291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.825784 10291 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.827363 7429 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.828843 10197 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.829169 7298 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.829349 7153 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:39.829623 7498 consensus_peers.cc:485] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 110ac05b2e8c4594a03225ab2c1ad56d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.832769 7298 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:39.833743 9972 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:39.833874 7431 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.850396 9972 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.853495 7298 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.853844 7156 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.862298 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.865600 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.866735 7431 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:39.869848 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 13:59:39.870997 7367 consensus_peers.cc:485] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c74f828c28c64097ab4bf5ec66459d2f. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:39.871136 7367 consensus_peers.cc:485] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c74f828c28c64097ab4bf5ec66459d2f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.871834 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.873739 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.874717 7693 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.874754 7298 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.875283 10298 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.875487 10349 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.878386 10298 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.881503 7149 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.882330 7298 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.883563 7429 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:39.887172 7693 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.890997 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.904561 7429 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:39.906986 7561 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.907563 8919 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.909667 8919 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.910239 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:39.910120 7561 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
W20240503 13:59:39.911424 7367 consensus_peers.cc:485] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 902cf3db3581465c83170c4b13b3bffb. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:39.911947 7367 consensus_peers.cc:485] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 902cf3db3581465c83170c4b13b3bffb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.913833 7149 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.923581 7298 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.924196 10489 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.925634 7298 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:39.939364 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.965669 7430 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
W20240503 13:59:39.967576 7367 consensus_peers.cc:485] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fca185e6c04648c5aa4f4ad51b0e1a39. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.967970 7561 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 13:59:39.968408 7367 consensus_peers.cc:485] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fca185e6c04648c5aa4f4ad51b0e1a39. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.969091 10489 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.970968 10489 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.973136 7561 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.973819 7693 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.974258 10299 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.974756 7156 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:39.975296 7691 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:39.983829 7430 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
W20240503 13:59:39.986987 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2212cf72b6ee4ee5b086455a75482e3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.987027 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 13:59:39.987113 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2212cf72b6ee4ee5b086455a75482e3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.988639 7561 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.989362 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.989948 7693 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:39.990427 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.990684 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:39.991461 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:39.991606 10299 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.992980 7691 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.994424 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:39.994453 7561 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:39.997216 7153 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.001494 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:40.004510 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:40.013453 10554 ts_tablet_manager.cc:902] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.014406 10554 tablet_copy_client.cc:308] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:40.018191 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:40.019668 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet b23f9eeeb4af45b7b679b70514e247b8 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:44957)
I20240503 13:59:40.019865 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet b23f9eeeb4af45b7b679b70514e247b8 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:44957: session id = da390a1a407a4f30b217b3d60fba563f-b23f9eeeb4af45b7b679b70514e247b8
I20240503 13:59:40.021418 7710 tablet_copy_source_session.cc:205] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.031443 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23f9eeeb4af45b7b679b70514e247b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.041618 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.047757 10563 ts_tablet_manager.cc:902] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.048768 10563 tablet_copy_client.cc:308] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:40.049762 10560 ts_tablet_manager.cc:902] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.051294 10560 tablet_copy_client.cc:308] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:40.055253 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 491ee8399ef04ae7b93e3228fef05eb0 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:40203)
I20240503 13:59:40.055423 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 491ee8399ef04ae7b93e3228fef05eb0 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:40203: session id = c5c3042421284f028fa512b7876da405-491ee8399ef04ae7b93e3228fef05eb0
I20240503 13:59:40.056900 7582 tablet_copy_source_session.cc:205] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.061260 10563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ee8399ef04ae7b93e3228fef05eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.063279 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:40.066370 10561 ts_tablet_manager.cc:902] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.067158 10561 tablet_copy_client.cc:308] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:40.068173 10562 ts_tablet_manager.cc:902] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:40.069408 10545 ts_tablet_manager.cc:902] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.070066 10545 tablet_copy_client.cc:308] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:40.070645 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:44957)
I20240503 13:59:40.070683 10563 tablet_copy_client.cc:783] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.070750 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 9445bc10fdcf4e9b8d9167ba243bfda5 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:44957: session id = da390a1a407a4f30b217b3d60fba563f-9445bc10fdcf4e9b8d9167ba243bfda5
I20240503 13:59:40.071039 10563 tablet_copy_client.cc:647] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.071761 7710 tablet_copy_source_session.cc:205] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.072940 10563 tablet_copy_client.cc:523] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.075124 10545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9445bc10fdcf4e9b8d9167ba243bfda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.075201 10563 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.077703 10545 tablet_copy_client.cc:783] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.078020 10545 tablet_copy_client.cc:647] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.078408 10554 tablet_copy_client.cc:783] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.078543 10554 tablet_copy_client.cc:647] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.082983 10563 log.cc:824] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:40.083247 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet e0b7095bdf3549fb83b3573a8e65deee from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:54357)
I20240503 13:59:40.083374 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet e0b7095bdf3549fb83b3573a8e65deee from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:54357: session id = 476f53c7e40c4a0b86ffbd818aea818d-e0b7095bdf3549fb83b3573a8e65deee
I20240503 13:59:40.083312 10590 ts_tablet_manager.cc:902] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.083863 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 63552cb8c9c94a07b2a0b157402efa7a from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:45115)
I20240503 13:59:40.083961 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 63552cb8c9c94a07b2a0b157402efa7a from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:45115: session id = 708734e6729441fcb2c5952fa47771d9-63552cb8c9c94a07b2a0b157402efa7a
I20240503 13:59:40.084372 7710 tablet_copy_source_session.cc:205] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.084862 7582 tablet_copy_source_session.cc:205] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.085844 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:40.088357 10560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63552cb8c9c94a07b2a0b157402efa7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.089409 10561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b7095bdf3549fb83b3573a8e65deee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.090514 10562 tablet_copy_client.cc:308] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:40.095312 10590 tablet_copy_client.cc:308] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:40.097788 10561 tablet_copy_client.cc:783] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.098124 10561 tablet_copy_client.cc:647] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.098276 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:45115)
I20240503 13:59:40.098393 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 8aff688eaad64d149bdc1c8ab12f4ca1 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:45115: session id = 708734e6729441fcb2c5952fa47771d9-8aff688eaad64d149bdc1c8ab12f4ca1
I20240503 13:59:40.099321 7582 tablet_copy_source_session.cc:205] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.100257 10590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aff688eaad64d149bdc1c8ab12f4ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.100483 10561 tablet_copy_client.cc:523] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.102566 10560 tablet_copy_client.cc:783] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.102847 10560 tablet_copy_client.cc:647] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.102845 10561 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.108129 10560 tablet_copy_client.cc:523] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.110350 10560 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.112394 10590 tablet_copy_client.cc:783] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.112499 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:40.113910 7430 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.114059 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 78bb6d55e8b44732b32da55f00071b21 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:38323)
I20240503 13:59:40.114213 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 78bb6d55e8b44732b32da55f00071b21 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:38323: session id = 708734e6729441fcb2c5952fa47771d9-78bb6d55e8b44732b32da55f00071b21
I20240503 13:59:40.115028 10590 tablet_copy_client.cc:647] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.115696 7320 tablet_copy_source_session.cc:205] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.117022 10590 tablet_copy_client.cc:523] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.118270 7561 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.118697 10349 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.119562 10590 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.121358 7429 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.123409 7298 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:40.124058 10349 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:40.124100 7691 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:40.124508 10349 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:40.126034 7693 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.126403 10299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.126675 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bb6d55e8b44732b32da55f00071b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.128999 10562 tablet_copy_client.cc:783] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.128882 7299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.129170 10562 tablet_copy_client.cc:647] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.131043 10489 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.131204 7156 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.132212 10530 ts_tablet_manager.cc:902] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.132768 10530 tablet_copy_client.cc:308] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
W20240503 13:59:40.135638 7367 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b7b43742e34431af0c16d320b35dd7. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:40.135756 7367 consensus_peers.cc:485] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1a85f321bfa4faf9b48cb319221fb94. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:40.135823 7367 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b7b43742e34431af0c16d320b35dd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.136469 7693 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
W20240503 13:59:40.138022 7367 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b7b43742e34431af0c16d320b35dd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.138732 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet e0b7095bdf3549fb83b3573a8e65deee from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:43367)
I20240503 13:59:40.138834 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet e0b7095bdf3549fb83b3573a8e65deee from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:43367: session id = c5c3042421284f028fa512b7876da405-e0b7095bdf3549fb83b3573a8e65deee
I20240503 13:59:40.139776 7710 tablet_copy_source_session.cc:205] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.141113 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.145560 10530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b7095bdf3549fb83b3573a8e65deee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.145870 10560 log.cc:824] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:40.146485 10603 ts_tablet_manager.cc:902] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:40.148701 10561 log.cc:824] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:40.150259 10603 tablet_copy_client.cc:308] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:40.150961 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet f73a5db0cae647d4b3375e6d42358307 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:38323)
I20240503 13:59:40.151079 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet f73a5db0cae647d4b3375e6d42358307 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:38323: session id = 708734e6729441fcb2c5952fa47771d9-f73a5db0cae647d4b3375e6d42358307
I20240503 13:59:40.151342 10530 tablet_copy_client.cc:783] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.151511 10530 tablet_copy_client.cc:647] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.152056 7320 tablet_copy_source_session.cc:205] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.154292 10603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73a5db0cae647d4b3375e6d42358307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.157153 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.157109 10603 tablet_copy_client.cc:783] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.157331 10603 tablet_copy_client.cc:647] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.157586 10605 ts_tablet_manager.cc:902] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.158231 10605 tablet_copy_client.cc:308] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:40.158921 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet b23f9eeeb4af45b7b679b70514e247b8 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:43367)
I20240503 13:59:40.159042 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet b23f9eeeb4af45b7b679b70514e247b8 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:43367: session id = c5c3042421284f028fa512b7876da405-b23f9eeeb4af45b7b679b70514e247b8
I20240503 13:59:40.160059 7710 tablet_copy_source_session.cc:205] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.163463 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.165362 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:40.174784 10562 tablet_copy_client.cc:523] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.175043 10603 tablet_copy_client.cc:523] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.178017 10562 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.181121 10603 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.182871 10605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23f9eeeb4af45b7b679b70514e247b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.184712 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.185592 10605 tablet_copy_client.cc:783] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.185827 10605 tablet_copy_client.cc:647] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.201049 10560 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.205811 10621 ts_tablet_manager.cc:902] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:40.206624 10621 tablet_copy_client.cc:308] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:40.213400 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 14304180134241c792ec1f23ff89f94c from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:56397)
I20240503 13:59:40.213548 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 14304180134241c792ec1f23ff89f94c from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:56397: session id = da390a1a407a4f30b217b3d60fba563f-14304180134241c792ec1f23ff89f94c
I20240503 13:59:40.214578 7320 tablet_copy_source_session.cc:205] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.215960 10554 tablet_copy_client.cc:523] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.217342 7299 consensus_queue.cc:237] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.217833 10560 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:40.218454 10554 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.218732 10560 ts_tablet_manager.cc:1367] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.109s user 0.036s sys 0.000s
I20240503 13:59:40.219192 7430 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.219879 7693 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.220265 10385 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.220955 10385 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:40.223521 7236 consensus_peers.cc:485] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b0fd771f844fcab38aec069b9abd00. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.234522 10545 tablet_copy_client.cc:523] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.236037 10623 ts_tablet_manager.cc:902] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.236321 10545 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.236851 10623 tablet_copy_client.cc:308] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:40.237600 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet ff246d9b2c594282bf2e99d2fd2b617d from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:44957)
I20240503 13:59:40.237730 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet ff246d9b2c594282bf2e99d2fd2b617d from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:44957: session id = da390a1a407a4f30b217b3d60fba563f-ff246d9b2c594282bf2e99d2fd2b617d
I20240503 13:59:40.238772 7710 tablet_copy_source_session.cc:205] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.239157 10560 raft_consensus.cc:348] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.239609 10560 raft_consensus.cc:729] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:40.239742 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff246d9b2c594282bf2e99d2fd2b617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.240085 10560 consensus_queue.cc:260] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.240433 10545 log.cc:824] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:40.241045 10605 tablet_copy_client.cc:523] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.242282 10623 tablet_copy_client.cc:783] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.242483 10623 tablet_copy_client.cc:647] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.242573 10605 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.242750 10628 ts_tablet_manager.cc:902] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:40.243279 10628 tablet_copy_client.cc:308] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:40.243543 10560 ts_tablet_manager.cc:1398] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20240503 13:59:40.244035 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 4582c127b7d34e08a68ea9094de11f19 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:43367)
I20240503 13:59:40.244159 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 4582c127b7d34e08a68ea9094de11f19 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:43367: session id = c5c3042421284f028fa512b7876da405-4582c127b7d34e08a68ea9094de11f19
I20240503 13:59:40.245183 7710 tablet_copy_source_session.cc:205] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.245028 10385 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.245311 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-63552cb8c9c94a07b2a0b157402efa7a received from {username='slave'} at 127.0.99.134:45115
I20240503 13:59:40.245414 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-63552cb8c9c94a07b2a0b157402efa7a on tablet 63552cb8c9c94a07b2a0b157402efa7a with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:40.247364 7430 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.250077 7153 catalog_manager.cc:5484] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.254195 7693 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.259495 7430 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
W20240503 13:59:40.261179 7367 consensus_peers.cc:485] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2cabe02dbf4580b53875aaa09aaffa. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.261291 7299 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.261725 7562 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.261752 10349 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.262459 10349 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.264089 10623 tablet_copy_client.cc:523] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.264603 10489 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:40.265089 7562 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:40.267238 7153 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.271188 10563 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.271585 10563 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.271950 10563 ts_tablet_manager.cc:1367] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.197s user 0.035s sys 0.002s
I20240503 13:59:40.279587 7562 consensus_queue.cc:237] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.281359 7300 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.282128 10201 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:40.282713 7498 consensus_peers.cc:485] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86634d81241c44cebcb602a6982f4244. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.282914 10563 raft_consensus.cc:348] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.283213 10563 raft_consensus.cc:729] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.283614 10563 consensus_queue.cc:260] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.284132 7693 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.284617 10201 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.285216 10590 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.285710 10590 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:40.285943 10590 ts_tablet_manager.cc:1367] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.166s user 0.025s sys 0.003s
I20240503 13:59:40.287019 10590 raft_consensus.cc:348] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.287146 10590 raft_consensus.cc:729] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:40.287243 10590 consensus_queue.cc:260] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.287317 10201 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.288189 10590 ts_tablet_manager.cc:1398] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:40.289484 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-8aff688eaad64d149bdc1c8ab12f4ca1 received from {username='slave'} at 127.0.99.134:45115
I20240503 13:59:40.289505 10563 ts_tablet_manager.cc:1398] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 13:59:40.289559 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-8aff688eaad64d149bdc1c8ab12f4ca1 on tablet 8aff688eaad64d149bdc1c8ab12f4ca1 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:40.290762 7693 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.291237 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session c5c3042421284f028fa512b7876da405-491ee8399ef04ae7b93e3228fef05eb0 received from {username='slave'} at 127.0.99.135:40203
I20240503 13:59:40.291316 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session c5c3042421284f028fa512b7876da405-491ee8399ef04ae7b93e3228fef05eb0 on tablet 491ee8399ef04ae7b93e3228fef05eb0 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.291539 7153 catalog_manager.cc:5484] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.298269 10603 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.298735 10603 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:40.299072 10603 ts_tablet_manager.cc:1367] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.118s user 0.026s sys 0.002s
I20240503 13:59:40.299635 10603 raft_consensus.cc:348] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.299898 10603 raft_consensus.cc:729] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:40.300105 10603 consensus_queue.cc:260] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.300467 10603 ts_tablet_manager.cc:1398] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:40.301172 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-f73a5db0cae647d4b3375e6d42358307 received from {username='slave'} at 127.0.99.134:38323
I20240503 13:59:40.301328 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-f73a5db0cae647d4b3375e6d42358307 on tablet f73a5db0cae647d4b3375e6d42358307 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:40.305258 10623 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.306699 10621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14304180134241c792ec1f23ff89f94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.307050 7562 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
W20240503 13:59:40.309000 7498 consensus_peers.cc:485] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa9fb136138439b8f694ac95dbfa2e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.309839 10621 tablet_copy_client.cc:783] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.309929 7430 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.310504 10628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4582c127b7d34e08a68ea9094de11f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.310526 7693 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.310402 10349 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.310894 9670 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.311269 9670 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.312655 7561 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.313660 7156 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.314371 10628 tablet_copy_client.cc:783] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.315042 10628 tablet_copy_client.cc:647] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.315317 10604 ts_tablet_manager.cc:902] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.315941 10604 tablet_copy_client.cc:308] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:40.316442 7691 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.321763 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:50181)
I20240503 13:59:40.321882 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 8aff688eaad64d149bdc1c8ab12f4ca1 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:50181: session id = 476f53c7e40c4a0b86ffbd818aea818d-8aff688eaad64d149bdc1c8ab12f4ca1
I20240503 13:59:40.322876 7582 tablet_copy_source_session.cc:205] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.324411 10604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aff688eaad64d149bdc1c8ab12f4ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.325497 9972 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.326498 7430 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.328644 10603 ts_tablet_manager.cc:902] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.328796 7153 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.329275 10603 tablet_copy_client.cc:308] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:40.332762 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet ade40ddaf0304b1a9e94db6630175c0e from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:45115)
I20240503 13:59:40.332870 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet ade40ddaf0304b1a9e94db6630175c0e from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:45115: session id = 708734e6729441fcb2c5952fa47771d9-ade40ddaf0304b1a9e94db6630175c0e
I20240503 13:59:40.333846 7582 tablet_copy_source_session.cc:205] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.334903 10603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade40ddaf0304b1a9e94db6630175c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.341393 10604 tablet_copy_client.cc:783] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.341424 10603 tablet_copy_client.cc:783] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.341598 10603 tablet_copy_client.cc:647] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.342052 7693 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.343336 10621 tablet_copy_client.cc:647] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.315364 10530 tablet_copy_client.cc:523] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.344071 10604 tablet_copy_client.cc:647] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.345543 10530 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.331389 10563 ts_tablet_manager.cc:902] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.346890 10563 tablet_copy_client.cc:308] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.348551 7300 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.350602 10604 tablet_copy_client.cc:523] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.352233 10621 tablet_copy_client.cc:523] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.353528 10604 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.355278 10621 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.355656 10603 tablet_copy_client.cc:523] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.357034 10603 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.358208 10628 tablet_copy_client.cc:523] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.358611 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet c74f828c28c64097ab4bf5ec66459d2f from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:40.358776 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet c74f828c28c64097ab4bf5ec66459d2f from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-c74f828c28c64097ab4bf5ec66459d2f
I20240503 13:59:40.359848 10628 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.360319 7451 tablet_copy_source_session.cc:205] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.361543 10563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74f828c28c64097ab4bf5ec66459d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.363618 10563 tablet_copy_client.cc:783] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.363747 10563 tablet_copy_client.cc:647] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.365713 10563 tablet_copy_client.cc:523] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.366994 10563 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.373489 9286 raft_consensus.cc:1203] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.375187 10561 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.375519 10561 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.375887 10561 ts_tablet_manager.cc:1367] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.273s user 0.037s sys 0.000s
I20240503 13:59:40.382161 10689 ts_tablet_manager.cc:902] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:40.382809 10689 tablet_copy_client.cc:308] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:40.387869 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:58407)
I20240503 13:59:40.388010 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet f3eaa7e5f641436a8f0cfdaf83443ca1 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:58407: session id = 476f53c7e40c4a0b86ffbd818aea818d-f3eaa7e5f641436a8f0cfdaf83443ca1
I20240503 13:59:40.388994 7320 tablet_copy_source_session.cc:205] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.395112 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3eaa7e5f641436a8f0cfdaf83443ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.395658 7430 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.396554 10563 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.396886 10707 ts_tablet_manager.cc:902] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.396939 10563 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.397141 10563 ts_tablet_manager.cc:1367] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20240503 13:59:40.397603 10707 tablet_copy_client.cc:308] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
W20240503 13:59:40.397835 7367 consensus_peers.cc:485] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2cabe02dbf4580b53875aaa09aaffa. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.397789 10563 raft_consensus.cc:348] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.398155 10563 raft_consensus.cc:729] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.398161 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 110ac05b2e8c4594a03225ab2c1ad56d from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:50181)
I20240503 13:59:40.398222 7561 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.398254 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 110ac05b2e8c4594a03225ab2c1ad56d from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:50181: session id = 476f53c7e40c4a0b86ffbd818aea818d-110ac05b2e8c4594a03225ab2c1ad56d
I20240503 13:59:40.398278 10563 consensus_queue.cc:260] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.398566 10563 ts_tablet_manager.cc:1398] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.398663 10706 ts_tablet_manager.cc:902] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.398728 10298 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.399253 7582 tablet_copy_source_session.cc:205] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.399245 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-c74f828c28c64097ab4bf5ec66459d2f received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:40.399330 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-c74f828c28c64097ab4bf5ec66459d2f on tablet c74f828c28c64097ab4bf5ec66459d2f with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.399574 10706 tablet_copy_client.cc:308] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.400169 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 902cf3db3581465c83170c4b13b3bffb from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:40.400261 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 902cf3db3581465c83170c4b13b3bffb from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-902cf3db3581465c83170c4b13b3bffb
W20240503 13:59:40.400466 7367 consensus_peers.cc:485] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2cabe02dbf4580b53875aaa09aaffa. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.400605 10298 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.401125 7451 tablet_copy_source_session.cc:205] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.402899 7561 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.403508 7153 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.405606 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110ac05b2e8c4594a03225ab2c1ad56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.406024 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902cf3db3581465c83170c4b13b3bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.406402 10689 tablet_copy_client.cc:783] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.406607 10689 tablet_copy_client.cc:647] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.410790 10545 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.411293 10545 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:40.411880 10545 ts_tablet_manager.cc:1367] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.176s user 0.030s sys 0.002s
I20240503 13:59:40.412425 7430 consensus_queue.cc:237] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.414535 7693 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:40.414589 7367 consensus_peers.cc:485] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dffd8718851475099a6696e4a2b8901. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.414769 10689 tablet_copy_client.cc:523] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.414973 10299 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.415434 7561 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.415874 10299 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.416774 10689 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.423821 10545 raft_consensus.cc:348] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.426753 10603 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.426990 10545 raft_consensus.cc:729] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:40.427488 10545 consensus_queue.cc:260] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.428709 10545 ts_tablet_manager.cc:1398] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240503 13:59:40.430617 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-9445bc10fdcf4e9b8d9167ba243bfda5 received from {username='slave'} at 127.0.99.136:44957
I20240503 13:59:40.430724 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-9445bc10fdcf4e9b8d9167ba243bfda5 on tablet 9445bc10fdcf4e9b8d9167ba243bfda5 with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.431699 10603 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:40.432606 10603 ts_tablet_manager.cc:1367] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.076s user 0.024s sys 0.004s
I20240503 13:59:40.433234 10603 raft_consensus.cc:348] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.434516 7300 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.435319 8919 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.435771 10707 tablet_copy_client.cc:783] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.435909 10706 tablet_copy_client.cc:783] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.436146 10707 tablet_copy_client.cc:647] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.436342 10706 tablet_copy_client.cc:647] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.433441 10603 raft_consensus.cc:729] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:40.436501 10603 consensus_queue.cc:260] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.436610 7300 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:40.436885 10603 ts_tablet_manager.cc:1398] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:40.438153 7300 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:40.440296 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-ade40ddaf0304b1a9e94db6630175c0e received from {username='slave'} at 127.0.99.134:45115
I20240503 13:59:40.440387 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-ade40ddaf0304b1a9e94db6630175c0e on tablet ade40ddaf0304b1a9e94db6630175c0e with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:40.440479 10706 tablet_copy_client.cc:523] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.443038 7300 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
W20240503 13:59:40.444603 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14304180134241c792ec1f23ff89f94c. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:40.445019 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.445264 7430 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.445638 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.445771 7691 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.446262 10142 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.447304 8919 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.448506 10707 tablet_copy_client.cc:523] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.449223 10142 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.450340 7561 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.450440 7691 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.450546 7156 catalog_manager.cc:5484] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.452998 7153 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.455559 9156 raft_consensus.cc:1203] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.456485 7430 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.460621 10707 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.460790 7693 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.463110 7430 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
W20240503 13:59:40.465374 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2212cf72b6ee4ee5b086455a75482e3b. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:40.465516 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2212cf72b6ee4ee5b086455a75482e3b. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:40.465636 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2212cf72b6ee4ee5b086455a75482e3b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.466205 7693 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:40.466583 10298 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:40.467728 7561 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:40.468179 10349 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:40.468840 10298 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:40.470911 7430 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.472499 7156 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.465444 10737 ts_tablet_manager.cc:902] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
W20240503 13:59:40.476636 7367 consensus_peers.cc:485] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1a85f321bfa4faf9b48cb319221fb94. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.466249 10561 raft_consensus.cc:348] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.476891 10561 raft_consensus.cc:729] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.477054 10737 tablet_copy_client.cc:308] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.477484 10561 consensus_queue.cc:260] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
W20240503 13:59:40.477867 7367 consensus_peers.cc:485] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1a85f321bfa4faf9b48cb319221fb94. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.478341 7562 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.478739 10299 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.479094 10561 ts_tablet_manager.cc:1398] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.103s user 0.007s sys 0.000s
I20240503 13:59:40.479653 10605 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.479684 7693 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:40.479970 10605 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.480154 10605 ts_tablet_manager.cc:1367] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.238s user 0.019s sys 0.004s
I20240503 13:59:40.480619 10605 raft_consensus.cc:348] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.480743 10605 raft_consensus.cc:729] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.480845 10605 consensus_queue.cc:260] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.481365 7691 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.481734 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet c74f828c28c64097ab4bf5ec66459d2f from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:39047)
I20240503 13:59:40.481820 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet c74f828c28c64097ab4bf5ec66459d2f from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:39047: session id = 476f53c7e40c4a0b86ffbd818aea818d-c74f828c28c64097ab4bf5ec66459d2f
I20240503 13:59:40.482492 10562 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.482748 7451 tablet_copy_source_session.cc:205] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.482774 10562 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:40.483798 10349 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.484244 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74f828c28c64097ab4bf5ec66459d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.486025 10562 ts_tablet_manager.cc:1367] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.308s user 0.028s sys 0.000s
I20240503 13:59:40.486222 9156 raft_consensus.cc:1203] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.486582 10737 tablet_copy_client.cc:783] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.486733 10737 tablet_copy_client.cc:647] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.487529 7561 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:40.490636 10706 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.492244 10298 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.493297 7691 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.495210 7562 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.498046 7153 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.498690 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-e0b7095bdf3549fb83b3573a8e65deee received from {username='slave'} at 127.0.99.133:54357
I20240503 13:59:40.498761 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-e0b7095bdf3549fb83b3573a8e65deee on tablet e0b7095bdf3549fb83b3573a8e65deee with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.499107 10562 raft_consensus.cc:348] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.499374 10562 raft_consensus.cc:729] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:40.499538 10562 consensus_queue.cc:260] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.500555 10562 ts_tablet_manager.cc:1398] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 13:59:40.501132 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-78bb6d55e8b44732b32da55f00071b21 received from {username='slave'} at 127.0.99.134:38323
I20240503 13:59:40.501210 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-78bb6d55e8b44732b32da55f00071b21 on tablet 78bb6d55e8b44732b32da55f00071b21 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:40.510438 10605 ts_tablet_manager.cc:1398] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240503 13:59:40.511001 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session c5c3042421284f028fa512b7876da405-b23f9eeeb4af45b7b679b70514e247b8 received from {username='slave'} at 127.0.99.135:43367
I20240503 13:59:40.511096 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session c5c3042421284f028fa512b7876da405-b23f9eeeb4af45b7b679b70514e247b8 on tablet b23f9eeeb4af45b7b679b70514e247b8 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.516860 10545 ts_tablet_manager.cc:902] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.517323 10545 tablet_copy_client.cc:308] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.520131 10737 tablet_copy_client.cc:523] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.521787 10737 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.522815 10554 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.522722 7562 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.523330 10554 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:40.523568 10554 ts_tablet_manager.cc:1367] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.305s user 0.031s sys 0.000s
I20240503 13:59:40.524827 7691 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 13:59:40.525161 7498 consensus_peers.cc:485] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa9fb136138439b8f694ac95dbfa2e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.525315 7430 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.526294 9670 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.526577 9670 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.529186 9022 raft_consensus.cc:1203] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.530412 10197 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.532058 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 902cf3db3581465c83170c4b13b3bffb from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:40713)
I20240503 13:59:40.532160 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 902cf3db3581465c83170c4b13b3bffb from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:40713: session id = da390a1a407a4f30b217b3d60fba563f-902cf3db3581465c83170c4b13b3bffb
I20240503 13:59:40.532871 10628 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.533035 7451 tablet_copy_source_session.cc:205] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.533181 10628 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.533744 7691 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.533798 7156 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.534240 10545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902cf3db3581465c83170c4b13b3bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.536384 10545 tablet_copy_client.cc:783] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.536551 10545 tablet_copy_client.cc:647] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.536672 10628 ts_tablet_manager.cc:1367] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.177s user 0.028s sys 0.000s
I20240503 13:59:40.537034 7430 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.537591 10554 raft_consensus.cc:348] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.537719 10554 raft_consensus.cc:729] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:40.537819 10554 consensus_queue.cc:260] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.537909 10628 raft_consensus.cc:348] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.538043 10628 raft_consensus.cc:729] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.538141 10628 consensus_queue.cc:260] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.540488 10628 ts_tablet_manager.cc:1398] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240503 13:59:40.543675 7498 consensus_peers.cc:485] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa9fb136138439b8f694ac95dbfa2e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.544517 10554 ts_tablet_manager.cc:1398] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240503 13:59:40.545567 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-b23f9eeeb4af45b7b679b70514e247b8 received from {username='slave'} at 127.0.99.136:44957
I20240503 13:59:40.545650 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-b23f9eeeb4af45b7b679b70514e247b8 on tablet b23f9eeeb4af45b7b679b70514e247b8 with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.547106 10545 tablet_copy_client.cc:523] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.548753 10545 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.548904 10689 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.549166 10689 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.549336 10689 ts_tablet_manager.cc:1367] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.133s user 0.024s sys 0.001s
I20240503 13:59:40.550899 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session c5c3042421284f028fa512b7876da405-4582c127b7d34e08a68ea9094de11f19 received from {username='slave'} at 127.0.99.135:43367
I20240503 13:59:40.550998 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session c5c3042421284f028fa512b7876da405-4582c127b7d34e08a68ea9094de11f19 on tablet 4582c127b7d34e08a68ea9094de11f19 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.551002 10689 raft_consensus.cc:348] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.551206 10689 raft_consensus.cc:729] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.551352 10689 consensus_queue.cc:260] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.551803 10689 ts_tablet_manager.cc:1398] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:40.552371 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-f3eaa7e5f641436a8f0cfdaf83443ca1 received from {username='slave'} at 127.0.99.133:58407
I20240503 13:59:40.552453 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-f3eaa7e5f641436a8f0cfdaf83443ca1 on tablet f3eaa7e5f641436a8f0cfdaf83443ca1 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.557617 7300 consensus_queue.cc:237] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.559413 7562 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.559880 7691 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:40.559886 10751 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:40.560287 7236 consensus_peers.cc:485] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3622884b9b314ccca24422d8fa05360a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.560664 10751 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:40.562583 7562 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.561920 10751 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.567245 7156 catalog_manager.cc:5484] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:40.570353 10628 ts_tablet_manager.cc:902] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.570824 10628 tablet_copy_client.cc:308] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.571293 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet fca185e6c04648c5aa4f4ad51b0e1a39 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:40.571406 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet fca185e6c04648c5aa4f4ad51b0e1a39 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-fca185e6c04648c5aa4f4ad51b0e1a39
I20240503 13:59:40.571522 10554 ts_tablet_manager.cc:902] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.572166 10554 tablet_copy_client.cc:308] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.572309 7451 tablet_copy_source_session.cc:205] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.572938 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 33b7b43742e34431af0c16d320b35dd7 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:40713)
I20240503 13:59:40.573031 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 33b7b43742e34431af0c16d320b35dd7 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:40713: session id = da390a1a407a4f30b217b3d60fba563f-33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:40.573751 7451 tablet_copy_source_session.cc:205] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.574384 10628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca185e6c04648c5aa4f4ad51b0e1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.574790 10530 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.575065 10530 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.577416 10628 tablet_copy_client.cc:783] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.577620 10628 tablet_copy_client.cc:647] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.580186 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.583585 10562 ts_tablet_manager.cc:902] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.584059 10562 tablet_copy_client.cc:308] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.584352 10604 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.584673 10604 tablet_bootstrap.cc:492] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.584859 10604 ts_tablet_manager.cc:1367] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.231s user 0.026s sys 0.000s
I20240503 13:59:40.585472 10623 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.585703 10554 tablet_copy_client.cc:783] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.585868 10554 tablet_copy_client.cc:647] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.585891 10623 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:40.586210 10623 ts_tablet_manager.cc:1367] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.281s user 0.033s sys 0.000s
I20240503 13:59:40.586916 10623 raft_consensus.cc:348] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.587172 10623 raft_consensus.cc:729] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:40.587380 10623 consensus_queue.cc:260] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.588613 10623 ts_tablet_manager.cc:1398] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:40.590219 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-ff246d9b2c594282bf2e99d2fd2b617d received from {username='slave'} at 127.0.99.136:44957
I20240503 13:59:40.590304 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-ff246d9b2c594282bf2e99d2fd2b617d on tablet ff246d9b2c594282bf2e99d2fd2b617d with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.592589 10530 ts_tablet_manager.cc:1367] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.247s user 0.023s sys 0.006s
I20240503 13:59:40.594444 10530 raft_consensus.cc:348] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.595223 10530 raft_consensus.cc:729] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.595412 10530 consensus_queue.cc:260] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.595832 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet fca185e6c04648c5aa4f4ad51b0e1a39 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:47471)
I20240503 13:59:40.595947 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet fca185e6c04648c5aa4f4ad51b0e1a39 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:47471: session id = 708734e6729441fcb2c5952fa47771d9-fca185e6c04648c5aa4f4ad51b0e1a39
I20240503 13:59:40.596921 7451 tablet_copy_source_session.cc:205] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.597754 10530 ts_tablet_manager.cc:1398] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:40.598014 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca185e6c04648c5aa4f4ad51b0e1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.598846 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session c5c3042421284f028fa512b7876da405-e0b7095bdf3549fb83b3573a8e65deee received from {username='slave'} at 127.0.99.135:43367
I20240503 13:59:40.598920 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session c5c3042421284f028fa512b7876da405-e0b7095bdf3549fb83b3573a8e65deee on tablet e0b7095bdf3549fb83b3573a8e65deee with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.601476 10562 tablet_copy_client.cc:783] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.601678 10562 tablet_copy_client.cc:647] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.612429 7691 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:40.617966 9156 raft_consensus.cc:1203] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.621675 10628 tablet_copy_client.cc:523] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.622184 10706 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.622469 10706 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.622678 10706 ts_tablet_manager.cc:1367] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.164s user 0.026s sys 0.002s
I20240503 13:59:40.623179 10706 raft_consensus.cc:348] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.623268 10628 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.623309 10706 raft_consensus.cc:729] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.623397 10706 consensus_queue.cc:260] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.624353 10706 ts_tablet_manager.cc:1398] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:40.624944 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-902cf3db3581465c83170c4b13b3bffb received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:40.625027 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-902cf3db3581465c83170c4b13b3bffb on tablet 902cf3db3581465c83170c4b13b3bffb with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.629045 10621 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.629449 10621 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:40.629709 10621 ts_tablet_manager.cc:1367] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.274s user 0.024s sys 0.002s
I20240503 13:59:40.632926 10554 tablet_copy_client.cc:523] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.634733 10554 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.646723 10604 raft_consensus.cc:348] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.647080 10604 raft_consensus.cc:729] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.647284 10604 consensus_queue.cc:260] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:40.647341 10621 raft_consensus.cc:348] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.647490 10621 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:40.647662 10621 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.648257 10621 ts_tablet_manager.cc:1398] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240503 13:59:40.649497 9419 raft_consensus.cc:1203] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.649890 10604 ts_tablet_manager.cc:1398] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.065s user 0.004s sys 0.000s
I20240503 13:59:40.650879 9418 raft_consensus.cc:1203] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.652655 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-8aff688eaad64d149bdc1c8ab12f4ca1 received from {username='slave'} at 127.0.99.133:50181
I20240503 13:59:40.652748 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-8aff688eaad64d149bdc1c8ab12f4ca1 on tablet 8aff688eaad64d149bdc1c8ab12f4ca1 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.652786 10628 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.653098 10628 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.653321 10628 ts_tablet_manager.cc:1367] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20240503 13:59:40.653851 10628 raft_consensus.cc:348] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.654093 10628 raft_consensus.cc:729] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.654245 10628 consensus_queue.cc:260] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.655953 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-14304180134241c792ec1f23ff89f94c received from {username='slave'} at 127.0.99.136:56397
I20240503 13:59:40.656050 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-14304180134241c792ec1f23ff89f94c on tablet 14304180134241c792ec1f23ff89f94c with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.656267 10628 ts_tablet_manager.cc:1398] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:40.657447 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-fca185e6c04648c5aa4f4ad51b0e1a39 received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:40.657531 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-fca185e6c04648c5aa4f4ad51b0e1a39 on tablet fca185e6c04648c5aa4f4ad51b0e1a39 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.664325 9286 raft_consensus.cc:1203] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.669407 10554 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.669720 10554 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:40.669929 10554 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.000s
I20240503 13:59:40.670461 10554 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.670609 10554 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:40.670711 10554 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.671629 10554 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:40.672590 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-33b7b43742e34431af0c16d320b35dd7 received from {username='slave'} at 127.0.99.136:40713
I20240503 13:59:40.672689 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-33b7b43742e34431af0c16d320b35dd7 on tablet 33b7b43742e34431af0c16d320b35dd7 with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.674468 10707 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.675665 10628 ts_tablet_manager.cc:902] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.675997 10707 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.676069 10628 tablet_copy_client.cc:308] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.676226 10707 ts_tablet_manager.cc:1367] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.216s user 0.026s sys 0.000s
I20240503 13:59:40.676605 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 33b7b43742e34431af0c16d320b35dd7 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:40.676705 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 33b7b43742e34431af0c16d320b35dd7 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:40.676736 10707 raft_consensus.cc:348] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.676872 10707 raft_consensus.cc:729] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.676967 10707 consensus_queue.cc:260] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.677623 7451 tablet_copy_source_session.cc:205] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.678035 10707 ts_tablet_manager.cc:1398] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:40.678525 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-110ac05b2e8c4594a03225ab2c1ad56d received from {username='slave'} at 127.0.99.133:50181
I20240503 13:59:40.678606 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-110ac05b2e8c4594a03225ab2c1ad56d on tablet 110ac05b2e8c4594a03225ab2c1ad56d with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.678905 10628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.681026 10737 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.681187 10628 tablet_copy_client.cc:783] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.681331 10628 tablet_copy_client.cc:647] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.681363 10737 tablet_bootstrap.cc:492] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.681545 10737 ts_tablet_manager.cc:1367] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.160s user 0.028s sys 0.000s
I20240503 13:59:40.682297 10737 raft_consensus.cc:348] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.682572 10737 raft_consensus.cc:729] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.682695 10737 consensus_queue.cc:260] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.682864 10628 tablet_copy_client.cc:523] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.683064 10737 ts_tablet_manager.cc:1398] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.684252 10628 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:40.686056 10562 tablet_copy_client.cc:523] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.687642 10562 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.690431 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-c74f828c28c64097ab4bf5ec66459d2f received from {username='slave'} at 127.0.99.133:39047
I20240503 13:59:40.690524 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-c74f828c28c64097ab4bf5ec66459d2f on tablet c74f828c28c64097ab4bf5ec66459d2f with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.704809 10545 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.705140 10545 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:40.705339 10545 ts_tablet_manager.cc:1367] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.157s user 0.016s sys 0.007s
I20240503 13:59:40.705837 10545 raft_consensus.cc:348] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.722054 10545 raft_consensus.cc:729] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:40.722312 10545 consensus_queue.cc:260] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:40.722687 10545 ts_tablet_manager.cc:1398] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240503 13:59:40.723659 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-902cf3db3581465c83170c4b13b3bffb received from {username='slave'} at 127.0.99.136:40713
I20240503 13:59:40.723760 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-902cf3db3581465c83170c4b13b3bffb on tablet 902cf3db3581465c83170c4b13b3bffb with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.726444 9022 raft_consensus.cc:1203] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.730360 9156 raft_consensus.cc:1203] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.740612 10562 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.740855 10737 ts_tablet_manager.cc:902] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:40.741114 10562 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:40.741322 10737 tablet_copy_client.cc:308] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:40.741449 10562 ts_tablet_manager.cc:1367] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.054s user 0.027s sys 0.000s
I20240503 13:59:40.741868 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 86634d81241c44cebcb602a6982f4244 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:50181)
I20240503 13:59:40.742014 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 86634d81241c44cebcb602a6982f4244 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:50181: session id = 476f53c7e40c4a0b86ffbd818aea818d-86634d81241c44cebcb602a6982f4244
I20240503 13:59:40.742187 10562 raft_consensus.cc:348] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.742350 10562 raft_consensus.cc:729] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:40.742478 10562 consensus_queue.cc:260] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:40.742911 10562 ts_tablet_manager.cc:1398] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.743039 7582 tablet_copy_source_session.cc:205] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.743515 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-fca185e6c04648c5aa4f4ad51b0e1a39 received from {username='slave'} at 127.0.99.134:47471
I20240503 13:59:40.743595 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-fca185e6c04648c5aa4f4ad51b0e1a39 on tablet fca185e6c04648c5aa4f4ad51b0e1a39 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:40.743755 10628 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.743968 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86634d81241c44cebcb602a6982f4244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.744014 10628 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:40.744238 10628 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.060s user 0.022s sys 0.004s
I20240503 13:59:40.744598 10707 ts_tablet_manager.cc:902] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.744839 10628 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.744975 10628 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:40.745079 10628 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.745115 10707 tablet_copy_client.cc:308] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.745373 10628 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:40.745857 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 33b7b43742e34431af0c16d320b35dd7 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:39047)
I20240503 13:59:40.745957 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 33b7b43742e34431af0c16d320b35dd7 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:39047: session id = 476f53c7e40c4a0b86ffbd818aea818d-33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:40.746794 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-33b7b43742e34431af0c16d320b35dd7 received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:40.746865 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-33b7b43742e34431af0c16d320b35dd7 on tablet 33b7b43742e34431af0c16d320b35dd7 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:40.746897 7451 tablet_copy_source_session.cc:205] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.747939 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.748409 9286 raft_consensus.cc:1203] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.750339 10737 tablet_copy_client.cc:783] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.750842 10737 tablet_copy_client.cc:647] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.754475 10707 tablet_copy_client.cc:783] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.755069 10707 tablet_copy_client.cc:647] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.762778 9286 raft_consensus.cc:1203] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.766544 10737 tablet_copy_client.cc:523] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.768215 10737 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.772626 10707 tablet_copy_client.cc:523] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.774250 10707 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.792997 9022 raft_consensus.cc:1203] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.816340 10545 ts_tablet_manager.cc:902] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:40.816869 10545 tablet_copy_client.cc:308] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:40.817538 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet a4b0fd771f844fcab38aec069b9abd00 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:56397)
I20240503 13:59:40.817662 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet a4b0fd771f844fcab38aec069b9abd00 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:56397: session id = da390a1a407a4f30b217b3d60fba563f-a4b0fd771f844fcab38aec069b9abd00
I20240503 13:59:40.818711 7320 tablet_copy_source_session.cc:205] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.820037 10545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0fd771f844fcab38aec069b9abd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.822588 10545 tablet_copy_client.cc:783] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.822835 10545 tablet_copy_client.cc:647] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.824576 10737 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.824923 10737 tablet_bootstrap.cc:492] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.825129 10737 ts_tablet_manager.cc:1367] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.009s
I20240503 13:59:40.825675 10737 raft_consensus.cc:348] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.825801 10737 raft_consensus.cc:729] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.825912 10737 consensus_queue.cc:260] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.826249 10737 ts_tablet_manager.cc:1398] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:40.827545 10545 tablet_copy_client.cc:523] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.829018 10545 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.832693 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-86634d81241c44cebcb602a6982f4244 received from {username='slave'} at 127.0.99.133:50181
I20240503 13:59:40.832814 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-86634d81241c44cebcb602a6982f4244 on tablet 86634d81241c44cebcb602a6982f4244 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.835578 9156 raft_consensus.cc:1203] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.838966 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:40.840816 9418 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:40.850687 9022 raft_consensus.cc:1203] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.851861 9022 raft_consensus.cc:1203] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:40.860599 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:40.865859 10707 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.870636 10707 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:40.871188 10707 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.097s user 0.027s sys 0.000s
I20240503 13:59:40.871860 10707 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.872454 10707 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:40.877640 10737 ts_tablet_manager.cc:902] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.878245 10737 tablet_copy_client.cc:308] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.880894 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 4e2cabe02dbf4580b53875aaa09aaffa from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:39047)
I20240503 13:59:40.881011 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 4e2cabe02dbf4580b53875aaa09aaffa from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:39047: session id = 476f53c7e40c4a0b86ffbd818aea818d-4e2cabe02dbf4580b53875aaa09aaffa
I20240503 13:59:40.882067 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.882246 7451 tablet_copy_source_session.cc:205] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.883704 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cabe02dbf4580b53875aaa09aaffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.885850 10737 tablet_copy_client.cc:783] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.886391 10737 tablet_copy_client.cc:647] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.892278 10737 tablet_copy_client.cc:523] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.893941 10737 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:40.898128 10707 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:40.898571 10707 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240503 13:59:40.902361 7451 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-33b7b43742e34431af0c16d320b35dd7 received from {username='slave'} at 127.0.99.133:39047
I20240503 13:59:40.902455 7451 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-33b7b43742e34431af0c16d320b35dd7 on tablet 33b7b43742e34431af0c16d320b35dd7 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.904743 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:40.906783 9418 raft_consensus.cc:1203] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.909041 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.910097 10562 ts_tablet_manager.cc:902] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.910669 10562 tablet_copy_client.cc:308] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.912218 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 4dffd8718851475099a6696e4a2b8901 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:47471)
I20240503 13:59:40.912358 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 4dffd8718851475099a6696e4a2b8901 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:47471: session id = 708734e6729441fcb2c5952fa47771d9-4dffd8718851475099a6696e4a2b8901
I20240503 13:59:40.913381 7451 tablet_copy_source_session.cc:205] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.914541 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffd8718851475099a6696e4a2b8901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.917616 10562 tablet_copy_client.cc:783] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.917798 10562 tablet_copy_client.cc:647] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.918569 10628 ts_tablet_manager.cc:902] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.919030 10628 tablet_copy_client.cc:308] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.921833 10706 ts_tablet_manager.cc:902] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.922340 10706 tablet_copy_client.cc:308] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.923331 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 4e2cabe02dbf4580b53875aaa09aaffa from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:40.923431 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 4e2cabe02dbf4580b53875aaa09aaffa from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-4e2cabe02dbf4580b53875aaa09aaffa
I20240503 13:59:40.924438 7450 tablet_copy_source_session.cc:205] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.926076 9972 raft_consensus.cc:1050] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:40.926622 9972 consensus_queue.cc:237] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:40.928402 9286 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.929010 9286 raft_consensus.cc:1203] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.929355 10201 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.930905 7300 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.931313 9670 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.932394 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 2212cf72b6ee4ee5b086455a75482e3b from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:40.932523 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 2212cf72b6ee4ee5b086455a75482e3b from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-2212cf72b6ee4ee5b086455a75482e3b
I20240503 13:59:40.933501 7451 tablet_copy_source_session.cc:205] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.934093 10554 ts_tablet_manager.cc:902] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:40.934612 10554 tablet_copy_client.cc:308] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:40.934996 10628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2212cf72b6ee4ee5b086455a75482e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.935127 7451 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 2212cf72b6ee4ee5b086455a75482e3b from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:40713)
I20240503 13:59:40.935235 7451 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 2212cf72b6ee4ee5b086455a75482e3b from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:40713: session id = da390a1a407a4f30b217b3d60fba563f-2212cf72b6ee4ee5b086455a75482e3b
I20240503 13:59:40.936116 7451 tablet_copy_source_session.cc:205] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.937007 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2212cf72b6ee4ee5b086455a75482e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.937361 10628 tablet_copy_client.cc:783] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.937511 10628 tablet_copy_client.cc:647] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.938180 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cabe02dbf4580b53875aaa09aaffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.939599 10554 tablet_copy_client.cc:783] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.939762 10554 tablet_copy_client.cc:647] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.940196 10706 tablet_copy_client.cc:783] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.940349 10706 tablet_copy_client.cc:647] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.941373 7430 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:40.941896 9670 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:40.942752 10562 tablet_copy_client.cc:523] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.944334 10562 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:40.951665 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.962450 9287 raft_consensus.cc:1203] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:40.964360 10621 ts_tablet_manager.cc:902] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:40.964905 10621 tablet_copy_client.cc:308] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:40.966745 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 3622884b9b314ccca24422d8fa05360a from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:56397)
I20240503 13:59:40.966856 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 3622884b9b314ccca24422d8fa05360a from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:56397: session id = da390a1a407a4f30b217b3d60fba563f-3622884b9b314ccca24422d8fa05360a
I20240503 13:59:40.967877 7320 tablet_copy_source_session.cc:205] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.968930 10621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622884b9b314ccca24422d8fa05360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.971334 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:40.971395 10621 tablet_copy_client.cc:783] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.971556 10621 tablet_copy_client.cc:647] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.989604 10554 tablet_copy_client.cc:523] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.991287 10554 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:40.992542 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.000800 11037 ts_tablet_manager.cc:902] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:41.001546 11037 tablet_copy_client.cc:308] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:41.002966 9287 raft_consensus.cc:1203] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:41.004043 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 2212cf72b6ee4ee5b086455a75482e3b from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:47471)
I20240503 13:59:41.004173 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 2212cf72b6ee4ee5b086455a75482e3b from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:47471: session id = 708734e6729441fcb2c5952fa47771d9-2212cf72b6ee4ee5b086455a75482e3b
I20240503 13:59:41.005141 7450 tablet_copy_source_session.cc:205] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:41.015942 11037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2212cf72b6ee4ee5b086455a75482e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.017591 10291 raft_consensus.cc:1050] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.018218 10291 consensus_queue.cc:237] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.019898 7562 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.020637 7429 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.021282 10852 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.021936 10852 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.023149 10623 ts_tablet_manager.cc:902] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:41.023691 10623 tablet_copy_client.cc:308] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:41.024230 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet c1a85f321bfa4faf9b48cb319221fb94 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:40713)
I20240503 13:59:41.024380 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet c1a85f321bfa4faf9b48cb319221fb94 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:40713: session id = da390a1a407a4f30b217b3d60fba563f-c1a85f321bfa4faf9b48cb319221fb94
I20240503 13:59:41.025424 7450 tablet_copy_source_session.cc:205] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:41.026405 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a85f321bfa4faf9b48cb319221fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.026528 9418 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.026968 10082 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.028632 10623 tablet_copy_client.cc:783] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:41.028776 10623 tablet_copy_client.cc:647] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:41.029150 11037 tablet_copy_client.cc:783] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:41.029395 11037 tablet_copy_client.cc:647] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:41.030551 11037 tablet_copy_client.cc:523] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.031981 11037 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:41.034497 10623 tablet_copy_client.cc:523] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.035890 10623 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:41.038765 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.042426 9670 raft_consensus.cc:1050] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:41.042929 9670 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.046316 10386 raft_consensus.cc:1050] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:41.046885 10386 consensus_queue.cc:237] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.048897 7431 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.050151 10752 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.051245 9156 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.052803 10752 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.055142 9287 raft_consensus.cc:1203] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 13:59:41.055462 9418 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.055480 10852 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.056793 7300 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.058785 7154 catalog_manager.cc:5484] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.062041 7562 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.062544 10752 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.063328 9155 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.063527 9022 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.064126 10707 ts_tablet_manager.cc:902] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:41.064070 7562 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.064889 10707 tablet_copy_client.cc:308] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:41.065819 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet c1a85f321bfa4faf9b48cb319221fb94 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:39047)
I20240503 13:59:41.065932 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet c1a85f321bfa4faf9b48cb319221fb94 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:39047: session id = 476f53c7e40c4a0b86ffbd818aea818d-c1a85f321bfa4faf9b48cb319221fb94
I20240503 13:59:41.066927 7450 tablet_copy_source_session.cc:205] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:41.067410 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.069401 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a85f321bfa4faf9b48cb319221fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.071621 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.071981 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.072199 10707 tablet_copy_client.cc:783] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:41.072340 10707 tablet_copy_client.cc:647] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:41.075459 10707 tablet_copy_client.cc:523] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.075824 7428 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.076297 10201 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.077461 10707 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:41.079133 10628 tablet_copy_client.cc:523] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.079288 10737 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.080760 10628 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:41.087939 11075 ts_tablet_manager.cc:902] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:41.089037 10201 raft_consensus.cc:1013] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: attempt to promote peer 476f53c7e40c4a0b86ffbd818aea818d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 13:59:41.089943 10291 raft_consensus.cc:1050] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.090576 10291 consensus_queue.cc:237] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.092450 9418 raft_consensus.cc:1203] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:41.092772 7427 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.093448 9287 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.093962 11037 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.094157 11054 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.094369 11037 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:41.094583 11037 ts_tablet_manager.cc:1367] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.000s
I20240503 13:59:41.094544 10852 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.095191 11037 raft_consensus.cc:348] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:41.095405 11037 raft_consensus.cc:729] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:41.095571 7299 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.095551 11037 consensus_queue.cc:260] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:41.096040 10852 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.097723 11037 ts_tablet_manager.cc:1398] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:41.098819 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-2212cf72b6ee4ee5b086455a75482e3b received from {username='slave'} at 127.0.99.134:47471
I20240503 13:59:41.098901 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-2212cf72b6ee4ee5b086455a75482e3b on tablet 2212cf72b6ee4ee5b086455a75482e3b with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:41.099282 9418 raft_consensus.cc:1203] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 13:59:41.099445 11075 tablet_copy_client.cc:308] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:41.099877 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 14304180134241c792ec1f23ff89f94c from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:38323)
I20240503 13:59:41.099918 11037 ts_tablet_manager.cc:902] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:41.099969 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 14304180134241c792ec1f23ff89f94c from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:38323: session id = 708734e6729441fcb2c5952fa47771d9-14304180134241c792ec1f23ff89f94c
I20240503 13:59:41.100450 11037 tablet_copy_client.cc:308] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:41.100935 7320 tablet_copy_source_session.cc:205] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:41.100955 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet faa9fb136138439b8f694ac95dbfa2e7 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:45115)
I20240503 13:59:41.101047 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet faa9fb136138439b8f694ac95dbfa2e7 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:45115: session id = 708734e6729441fcb2c5952fa47771d9-faa9fb136138439b8f694ac95dbfa2e7
I20240503 13:59:41.102061 7582 tablet_copy_source_session.cc:205] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:41.102097 11075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14304180134241c792ec1f23ff89f94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.103819 10554 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.104158 11037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa9fb136138439b8f694ac95dbfa2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.105048 11075 tablet_copy_client.cc:783] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:41.105257 11075 tablet_copy_client.cc:647] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:41.105912 10197 raft_consensus.cc:1050] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:41.106395 11037 tablet_copy_client.cc:783] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:41.106436 10197 consensus_queue.cc:237] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.106546 11037 tablet_copy_client.cc:647] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:41.108059 7691 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.108057 9154 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.108480 9670 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.108798 7426 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.108934 9972 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.109112 10562 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.109351 10562 tablet_bootstrap.cc:492] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:41.109537 10562 ts_tablet_manager.cc:1367] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.165s user 0.025s sys 0.000s
I20240503 13:59:41.110080 10562 raft_consensus.cc:348] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:41.110180 10562 raft_consensus.cc:729] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:41.110149 9972 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.110370 10737 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:41.110561 10737 ts_tablet_manager.cc:1367] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.217s user 0.024s sys 0.003s
I20240503 13:59:41.111034 10737 raft_consensus.cc:348] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:41.111204 10737 raft_consensus.cc:729] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:41.111320 10737 consensus_queue.cc:260] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:41.111603 10737 ts_tablet_manager.cc:1398] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.112609 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-4e2cabe02dbf4580b53875aaa09aaffa received from {username='slave'} at 127.0.99.133:39047
I20240503 13:59:41.112697 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-4e2cabe02dbf4580b53875aaa09aaffa on tablet 4e2cabe02dbf4580b53875aaa09aaffa with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:41.121665 11075 tablet_copy_client.cc:523] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.124378 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.125191 11075 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:41.131227 10554 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:41.131492 10554 ts_tablet_manager.cc:1367] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.140s user 0.028s sys 0.001s
I20240503 13:59:41.132067 10554 raft_consensus.cc:348] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:41.132220 10554 raft_consensus.cc:729] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:41.132335 10554 consensus_queue.cc:260] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:41.133383 10562 consensus_queue.cc:260] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:41.134305 10562 ts_tablet_manager.cc:1398] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 13:59:41.134997 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-4dffd8718851475099a6696e4a2b8901 received from {username='slave'} at 127.0.99.134:47471
I20240503 13:59:41.135089 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-4dffd8718851475099a6696e4a2b8901 on tablet 4dffd8718851475099a6696e4a2b8901 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:41.136382 10554 ts_tablet_manager.cc:1398] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:41.137444 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-2212cf72b6ee4ee5b086455a75482e3b received from {username='slave'} at 127.0.99.136:40713
I20240503 13:59:41.137532 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-2212cf72b6ee4ee5b086455a75482e3b on tablet 2212cf72b6ee4ee5b086455a75482e3b with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:41.149564 10707 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.150101 10707 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:41.150395 10707 ts_tablet_manager.cc:1367] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.073s user 0.022s sys 0.005s
I20240503 13:59:41.150982 10707 raft_consensus.cc:348] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.151233 10707 raft_consensus.cc:729] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:41.151378 10707 consensus_queue.cc:260] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.151451 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.151844 10707 ts_tablet_manager.cc:1398] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:41.152015 10082 raft_consensus.cc:1050] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.152437 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-c1a85f321bfa4faf9b48cb319221fb94 received from {username='slave'} at 127.0.99.133:39047
I20240503 13:59:41.152519 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-c1a85f321bfa4faf9b48cb319221fb94 on tablet c1a85f321bfa4faf9b48cb319221fb94 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:41.152598 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.152551 10082 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.154819 9022 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.155108 9287 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.155305 10852 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.155603 10852 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.155814 7561 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.156294 10852 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.156948 7427 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.157815 11054 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.158547 10752 raft_consensus.cc:1050] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:41.159034 10752 consensus_queue.cc:237] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.160845 7425 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.161214 10142 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.162918 9154 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.163416 10142 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.164201 7562 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.164618 10142 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.166155 10554 ts_tablet_manager.cc:902] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:41.166694 10554 tablet_copy_client.cc:308] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:41.171519 11037 tablet_copy_client.cc:523] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.173151 11037 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:41.177290 10142 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.177906 7425 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.179553 7562 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.180583 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.180691 7157 catalog_manager.cc:5484] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.183985 9154 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.186973 7298 consensus_queue.cc:237] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.189134 7429 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.193418 7693 consensus_queue.cc:237] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.195937 10385 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.198482 9418 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.198729 7428 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.198664 10201 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.199025 10291 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.199111 9155 raft_consensus.cc:1203] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:41.200130 9155 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.200649 9670 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.201052 10291 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.201617 7428 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.202714 10623 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.202857 7425 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.203058 10623 tablet_bootstrap.cc:492] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:41.203256 10623 ts_tablet_manager.cc:1367] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.167s user 0.027s sys 0.000s
I20240503 13:59:41.203416 10142 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.203888 10623 raft_consensus.cc:348] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.204094 11054 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.204262 9153 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.204362 9286 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.205471 7431 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.206089 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.205471 7430 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.207159 7298 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.207497 10623 raft_consensus.cc:729] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:41.207641 10623 consensus_queue.cc:260] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.207988 10623 ts_tablet_manager.cc:1398] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:59:41.208173 7153 catalog_manager.cc:5484] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.210261 7154 catalog_manager.cc:5484] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.211433 9022 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.213292 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-c1a85f321bfa4faf9b48cb319221fb94 received from {username='slave'} at 127.0.99.136:40713
I20240503 13:59:41.213379 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-c1a85f321bfa4faf9b48cb319221fb94 on tablet c1a85f321bfa4faf9b48cb319221fb94 with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:41.214244 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:41.214716 10197 raft_consensus.cc:1050] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.214740 9418 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.215167 10197 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.217299 10291 raft_consensus.cc:1050] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.217716 10291 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.218958 7153 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 5 to 6, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.219429 7430 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.219997 7296 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.220506 10852 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.220749 7154 catalog_manager.cc:5484] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.221230 7431 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.221681 7298 consensus_queue.cc:237] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.221853 10852 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.224129 7429 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.224406 9670 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.225178 9418 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.225499 10385 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.225703 11054 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.227339 9022 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.227862 9670 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.228083 7562 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.230981 9156 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.231215 9152 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.230993 9151 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.231755 9670 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.232667 10142 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.232669 10385 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240503 13:59:41.238899 10752 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.239377 7428 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.240522 10142 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.242327 9151 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.242558 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78bb6d55e8b44732b32da55f00071b21 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.243705 10621 tablet_copy_client.cc:523] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.245242 10621 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:41.246488 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.247217 9972 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.248769 7157 catalog_manager.cc:5484] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.249534 7429 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.250815 7153 catalog_manager.cc:5484] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.252599 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.252976 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bb6d55e8b44732b32da55f00071b21 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:41.253579 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet f73a5db0cae647d4b3375e6d42358307 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:41.254524 9152 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.256196 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f73a5db0cae647d4b3375e6d42358307 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.258740 7153 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.265571 7297 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.266182 9670 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.268133 7297 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.269809 7297 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.270385 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.273468 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.280774 9152 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.283085 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.286340 10349 raft_consensus.cc:1050] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.287086 10349 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.287184 7561 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.288404 9022 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.289757 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.290356 7300 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.290887 9025 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.290894 9024 raft_consensus.cc:1203] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 13:59:41.291397 10299 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.291831 10299 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.291939 7693 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.292352 10349 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.293915 9286 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.293891 11054 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.294654 10299 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.295001 10142 raft_consensus.cc:1050] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.295459 10142 consensus_queue.cc:237] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.296492 7562 consensus_queue.cc:237] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.296911 7561 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.296838 7296 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.297294 10752 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.297353 9152 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.298024 9025 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.298148 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.298933 9285 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.299013 7692 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.299443 10752 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.299614 10197 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.300455 7429 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.300846 10197 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.302461 9025 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.302886 9972 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.304939 7428 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.306690 7157 catalog_manager.cc:5484] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.307799 9284 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.311473 10752 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.313218 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.317744 7690 consensus_queue.cc:237] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.322149 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 13:59:41.325091 11151 log.cc:924] Time spent T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240503 13:59:41.327996 9284 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.328198 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.329380 10082 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.329854 10082 raft_consensus.cc:1050] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.330384 10082 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.330842 9972 raft_consensus.cc:1050] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.331204 9972 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.332559 7296 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.333021 7296 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.333155 10201 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.333873 7562 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.334277 7300 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.334728 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.335301 10291 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.335597 10291 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.336483 9025 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.336827 9670 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.338413 9670 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.340055 7300 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.342123 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.342928 9418 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.343401 9283 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.343412 11054 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.344024 9152 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.344720 11054 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.345547 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet faa9fb136138439b8f694ac95dbfa2e7 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:33557)
I20240503 13:59:41.345649 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet faa9fb136138439b8f694ac95dbfa2e7 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:33557: session id = da390a1a407a4f30b217b3d60fba563f-faa9fb136138439b8f694ac95dbfa2e7
I20240503 13:59:41.345669 7424 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.346223 9670 raft_consensus.cc:1050] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:41.346629 7582 tablet_copy_source_session.cc:205] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:41.346771 9670 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.347652 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa9fb136138439b8f694ac95dbfa2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:41.347705 9670 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.348965 7296 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.349494 9152 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.350206 10201 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.350471 7423 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.350571 10201 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.350891 10197 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.351387 10554 tablet_copy_client.cc:783] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:41.351552 10554 tablet_copy_client.cc:647] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:41.352322 9023 raft_consensus.cc:1203] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:41.353401 7428 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.353843 11054 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.355252 7157 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.355872 11054 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.356662 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.358317 10752 raft_consensus.cc:1050] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.358795 10752 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:41.360407 7690 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.360891 10386 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:59:41.361765 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:41.362052 9419 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.362048 7410 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:41.362498 7422 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.362614 7560 consensus_queue.cc:237] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.363217 10386 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.364209 7428 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.364943 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4582c127b7d34e08a68ea9094de11f19 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.364969 9151 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.365000 9023 raft_consensus.cc:1261] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.365206 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.365348 10385 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.366178 9419 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.366264 7157 catalog_manager.cc:5484] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.367789 9283 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.368466 7690 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.369133 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 4582c127b7d34e08a68ea9094de11f19 with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.369570 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.372048 7157 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.373219 7428 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.374248 9151 raft_consensus.cc:1203] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:41.377527 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.369894 10197 consensus_queue.cc:1035] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.381381 9023 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.383934 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.386273 11157 tablet_replica.cc:330] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:41.386313 9670 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.386708 11157 raft_consensus.cc:2229] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.386937 11157 raft_consensus.cc:2258] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.387523 10554 tablet_copy_client.cc:523] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.388240 11157 ts_tablet_manager.cc:1875] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.388825 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.390025 9151 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.390545 9023 raft_consensus.cc:2932] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.390666 11157 ts_tablet_manager.cc:1888] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.390745 11157 log.cc:1196] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9445bc10fdcf4e9b8d9167ba243bfda5
I20240503 13:59:41.392693 7151 catalog_manager.cc:5484] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.394338 10554 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:41.398254 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aff688eaad64d149bdc1c8ab12f4ca1 with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.402302 10197 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.402802 7426 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.403913 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.404017 11183 tablet_replica.cc:330] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:41.404412 11160 raft_consensus.cc:1050] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.404580 11183 raft_consensus.cc:2229] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.404826 11183 raft_consensus.cc:2258] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.404903 11160 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.405890 9151 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.406461 7153 catalog_manager.cc:5484] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.407658 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.410924 9284 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.410908 9972 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.411516 10299 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.411985 10385 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.414173 7690 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.414394 7561 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.414470 7560 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.414602 11160 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.414503 9025 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.414924 10299 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.416309 7296 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.418177 9151 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.420184 9025 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.422607 7430 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.424095 7426 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.425040 10299 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.425393 7155 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.425843 7155 catalog_manager.cc:5484] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.426930 7691 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.429541 9972 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.430992 7560 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.431695 7429 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.432696 7296 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.433302 9025 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.433318 10201 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.433825 10197 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.437594 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 491ee8399ef04ae7b93e3228fef05eb0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.437734 9025 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.437427 7562 consensus_queue.cc:237] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.436939 9670 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.440769 7157 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.442308 9156 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.442780 9972 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.443745 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 110ac05b2e8c4594a03225ab2c1ad56d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.444115 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.444245 7296 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.445184 11183 ts_tablet_manager.cc:1875] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.445293 7151 catalog_manager.cc:5484] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.446866 10197 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.448328 11183 ts_tablet_manager.cc:1888] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:41.449368 9156 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.451426 7693 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.451982 7691 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.452481 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 491ee8399ef04ae7b93e3228fef05eb0 with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.452737 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63552cb8c9c94a07b2a0b157402efa7a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.451426 7692 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.452986 7155 catalog_manager.cc:5484] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.453754 10706 tablet_copy_client.cc:523] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:41.455349 10706 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:41.455667 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.456696 7151 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.458063 11156 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.459352 7691 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.459455 9419 raft_consensus.cc:1203] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:41.467038 9024 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.469772 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.484473 9024 raft_consensus.cc:1203] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:41.499781 10852 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.502084 9024 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.502631 7157 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.503363 7562 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.504019 10201 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.505237 7300 consensus_queue.cc:237] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.506251 9285 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.506804 9024 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.507031 7562 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.507246 10142 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.507418 10386 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.509277 10752 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.509536 7300 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.509842 9024 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.511102 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.511466 11183 log.cc:1196] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8aff688eaad64d149bdc1c8ab12f4ca1
I20240503 13:59:41.511445 7562 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.511655 9152 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.511991 7423 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.512403 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet 8aff688eaad64d149bdc1c8ab12f4ca1 (table rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]) successfully deleted
I20240503 13:59:41.451011 10299 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.514813 10299 raft_consensus.cc:1050] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.515180 10299 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.452874 7410 tablet_service.cc:1514] Processing DeleteTablet for tablet 110ac05b2e8c4594a03225ab2c1ad56d with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:41.467334 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 63552cb8c9c94a07b2a0b157402efa7a with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:41.470139 7425 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.518045 8919 raft_consensus.cc:1013] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: attempt to promote peer 476f53c7e40c4a0b86ffbd818aea818d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 13:59:41.518074 7156 catalog_manager.cc:5484] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.518379 7156 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.487052 10349 raft_consensus.cc:1050] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.519995 7691 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.520112 10349 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:41.520408 10299 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.521432 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:41.522471 7692 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.522560 7300 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.522938 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.523250 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.524093 10082 raft_consensus.cc:1050] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.524843 10082 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.526221 11200 tablet_replica.cc:330] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:41.526759 9287 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.526959 7562 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.527329 10852 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.528029 11200 raft_consensus.cc:2229] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.529058 9285 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.529546 9025 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.530210 9285 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.530659 9287 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.531420 11200 raft_consensus.cc:2258] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.532678 10852 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.534360 10852 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.535925 9025 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.536031 9285 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.536765 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.536731 11160 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.537086 9419 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.537617 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.537823 8919 raft_consensus.cc:1050] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.538245 8919 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.538578 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.540244 7295 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.540820 11054 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.542917 7560 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.543903 9283 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.544958 11200 ts_tablet_manager.cc:1875] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.544976 10349 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.546846 9419 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.546911 10349 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.547356 10349 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.547223 9025 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.548074 10349 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.549077 9024 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.549161 9419 raft_consensus.cc:1203] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 13:59:41.551491 10299 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.551887 10299 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.553937 7562 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.554549 10349 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.554915 10349 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.556077 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.555939 9286 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.558588 7691 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.560884 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.562667 9024 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.564960 9152 raft_consensus.cc:1203] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 13:59:41.565652 10082 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.566604 7296 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.567203 11160 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.568373 7299 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.569092 10852 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.569854 7299 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.571250 7560 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.572224 7297 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.573410 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet 9445bc10fdcf4e9b8d9167ba243bfda5 (table rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]) successfully deleted
I20240503 13:59:41.573722 11157 tablet_replica.cc:330] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:41.573874 11157 raft_consensus.cc:2229] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.574064 11157 raft_consensus.cc:2258] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.574497 7296 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.574923 11157 ts_tablet_manager.cc:1875] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.576185 7431 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.577404 7151 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.579173 9418 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.579638 9287 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.579919 10082 raft_consensus.cc:1050] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.580324 10082 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.582158 7560 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.582494 7154 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.582814 10298 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.583098 10852 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.585072 7296 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.585088 7561 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.585456 10852 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.586463 9287 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.587163 9284 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.587189 9416 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.587687 7690 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.588721 9418 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.589200 11183 tablet_replica.cc:330] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:41.589339 11183 raft_consensus.cc:2229] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.589486 11183 raft_consensus.cc:2258] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.590749 10852 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.591413 10852 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.591495 7154 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.594805 11184 tablet_replica.cc:330] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.595286 11184 raft_consensus.cc:2229] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.595587 11184 raft_consensus.cc:2258] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.601683 9287 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.603328 10852 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.604583 11183 ts_tablet_manager.cc:1875] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.604584 7154 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.605532 7299 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.606074 7560 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.608338 7423 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.608373 7431 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.609023 11183 ts_tablet_manager.cc:1888] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.609293 11183 log.cc:1196] T 110ac05b2e8c4594a03225ab2c1ad56d P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/110ac05b2e8c4594a03225ab2c1ad56d
I20240503 13:59:41.609843 7560 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.610210 7299 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.610447 10298 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.611601 9284 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.611852 9025 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.612452 7692 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.612493 9284 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.613030 10299 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.613130 7153 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.613315 10299 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.613500 10298 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.613903 10298 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.614727 9024 raft_consensus.cc:1203] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:41.616731 9287 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.619076 10298 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.619498 11184 ts_tablet_manager.cc:1875] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.619495 7157 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.621028 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fca185e6c04648c5aa4f4ad51b0e1a39 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.621407 7560 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.621927 11184 ts_tablet_manager.cc:1888] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.622030 11184 log.cc:1196] T 4582c127b7d34e08a68ea9094de11f19 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4582c127b7d34e08a68ea9094de11f19
I20240503 13:59:41.622596 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 4582c127b7d34e08a68ea9094de11f19 (table rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]) successfully deleted
I20240503 13:59:41.625718 7692 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.627923 9416 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.631870 11198 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.635003 10299 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.637323 11183 tablet_replica.cc:330] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:41.637558 11183 raft_consensus.cc:2229] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.637322 7692 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.637509 7300 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.638350 11183 raft_consensus.cc:2258] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.639609 11200 ts_tablet_manager.cc:1888] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.639974 11200 log.cc:1196] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f3eaa7e5f641436a8f0cfdaf83443ca1
I20240503 13:59:41.639974 7299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.640134 7560 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.641438 10349 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.641710 10298 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.642635 7157 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.643626 10349 raft_consensus.cc:1050] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:41.644016 10349 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:41.644769 7299 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.645951 7692 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.646510 7300 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.646926 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.647238 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.650398 9283 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.652984 9283 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.653075 7154 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.653667 7300 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.654079 11160 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.655359 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.657711 9283 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.658504 9024 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.659705 11183 ts_tablet_manager.cc:1875] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.661020 10298 raft_consensus.cc:1050] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.661427 10298 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.640852 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet f3eaa7e5f641436a8f0cfdaf83443ca1 (table rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]) successfully deleted
I20240503 13:59:41.663311 7562 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.663435 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet 110ac05b2e8c4594a03225ab2c1ad56d (table rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]) successfully deleted
I20240503 13:59:41.663583 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c74f828c28c64097ab4bf5ec66459d2f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.664489 7691 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.665326 7300 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.665810 9151 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.665796 11158 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.665699 7692 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.666224 7560 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.666360 9670 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.666599 10298 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.667570 9670 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.668072 7692 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.668293 9416 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.668488 7690 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.669061 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet c74f828c28c64097ab4bf5ec66459d2f with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:41.669077 7300 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.669618 10349 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.669736 11054 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.670253 9416 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.670185 7157 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER, now has a pending config: VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132), VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129), VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130), VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136), VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135), NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } } pending_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.670485 7300 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.670611 11054 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.670974 7560 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.671437 11054 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.670281 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ade40ddaf0304b1a9e94db6630175c0e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.672533 9024 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.672935 11170 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.673465 9151 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.674168 9285 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.674587 10082 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.674646 9283 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.675158 10082 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.675804 11183 ts_tablet_manager.cc:1888] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.676146 11170 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.676524 11183 log.cc:1196] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/63552cb8c9c94a07b2a0b157402efa7a
I20240503 13:59:41.678053 7562 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.678215 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff246d9b2c594282bf2e99d2fd2b617d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.678587 11054 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.678913 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet 63552cb8c9c94a07b2a0b157402efa7a (table rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]) successfully deleted
I20240503 13:59:41.678828 9416 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.678977 10852 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.680259 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b23f9eeeb4af45b7b679b70514e247b8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.680724 9285 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.680725 7561 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.683545 7149 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.684140 7153 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 5 to 6, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.686367 9136 tablet_service.cc:1514] Processing DeleteTablet for tablet fca185e6c04648c5aa4f4ad51b0e1a39 with delete_type TABLET_DATA_TOMBSTONED (TS 708734e6729441fcb2c5952fa47771d9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56318
I20240503 13:59:41.687116 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet ff246d9b2c594282bf2e99d2fd2b617d with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.687518 7157 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132), VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129), VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130), VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136), VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135), NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133). New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.688534 8919 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.689191 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:41.689905 11200 tablet_replica.cc:330] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:41.689949 9283 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.690090 11200 raft_consensus.cc:2229] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.690271 11200 raft_consensus.cc:2258] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.690843 7431 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.691059 7300 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.691318 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.693315 9415 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.693289 10349 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.694380 11170 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.694307 9283 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.694435 9416 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.695510 7149 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.695936 9283 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.696803 7562 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.697814 10349 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.699524 7431 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.699632 9416 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.701869 7423 consensus_queue.cc:237] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.702414 7149 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.703074 11170 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.703434 9283 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.704159 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet ade40ddaf0304b1a9e94db6630175c0e with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:41.704334 10349 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.704389 9023 raft_consensus.cc:1261] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.704654 11183 tablet_replica.cc:330] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:41.704804 11183 raft_consensus.cc:2229] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.704941 11183 raft_consensus.cc:2258] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.705921 10349 consensus_queue.cc:1035] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.705940 11183 ts_tablet_manager.cc:1875] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.706797 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0b7095bdf3549fb83b3573a8e65deee with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.707023 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.708151 7157 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.708813 9285 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.710294 9418 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.711040 11183 ts_tablet_manager.cc:1888] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.711110 11183 log.cc:1196] T ade40ddaf0304b1a9e94db6630175c0e P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ade40ddaf0304b1a9e94db6630175c0e
I20240503 13:59:41.712854 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet ade40ddaf0304b1a9e94db6630175c0e (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:41.713150 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet b23f9eeeb4af45b7b679b70514e247b8 with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.713816 7562 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.714273 7690 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:41.715883 9285 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.717062 11170 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.717705 11197 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.719347 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c74f828c28c64097ab4bf5ec66459d2f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.721596 7157 catalog_manager.cc:5484] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.721621 9285 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.722514 9024 raft_consensus.cc:2932] T c74f828c28c64097ab4bf5ec66459d2f P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.725282 11054 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.726771 7562 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.727216 11170 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.728070 9285 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.728559 7560 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:41.729929 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet c74f828c28c64097ab4bf5ec66459d2f with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.730322 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b23f9eeeb4af45b7b679b70514e247b8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.730808 11200 ts_tablet_manager.cc:1875] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.731036 9023 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.732093 9419 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.733549 9025 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:41.734071 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.735405 8919 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.735644 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.736302 9025 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.737437 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.737824 9025 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:41.716993 7154 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:41.741178 10706 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.744974 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.748104 7300 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:41.749434 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet e0b7095bdf3549fb83b3573a8e65deee with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:41.749691 11183 tablet_replica.cc:330] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:41.749877 11183 raft_consensus.cc:2229] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.750075 11183 pending_rounds.cc:62] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Trying to abort 1 pending ops.
I20240503 13:59:41.750156 11183 pending_rounds.cc:68] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023594623599366144 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "e0b7095bdf3549fb83b3573a8e65deee" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } } }
I20240503 13:59:41.750325 11183 raft_consensus.cc:2894] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 13:59:41.750252 7157 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.750383 11183 raft_consensus.cc:2866] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:41.750427 11183 raft_consensus.cc:2258] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.750792 7690 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.751616 10386 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.753618 10142 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.744994 10706 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:41.754585 10706 ts_tablet_manager.cc:1367] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.299s user 0.028s sys 0.000s
I20240503 13:59:41.755741 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14304180134241c792ec1f23ff89f94c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.756167 7690 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.756953 10706 raft_consensus.cc:348] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:41.757275 9398 tablet_service.cc:1514] Processing DeleteTablet for tablet b23f9eeeb4af45b7b679b70514e247b8 with delete_type TABLET_DATA_TOMBSTONED (TS da390a1a407a4f30b217b3d60fba563f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50322
I20240503 13:59:41.757737 10706 raft_consensus.cc:729] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:41.757915 10706 consensus_queue.cc:260] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
W20240503 13:59:41.758139 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:41.758452 10706 ts_tablet_manager.cc:1398] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:41.759091 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-4e2cabe02dbf4580b53875aaa09aaffa received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:41.759183 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-4e2cabe02dbf4580b53875aaa09aaffa on tablet 4e2cabe02dbf4580b53875aaa09aaffa with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:41.759711 8919 raft_consensus.cc:1050] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.760202 9417 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.760211 8919 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.760605 10386 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.763083 9025 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.763604 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.764943 7300 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.765388 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.765745 9417 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.766570 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.766682 9418 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.766937 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.767558 9285 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.768196 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.769811 7692 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.770671 11157 ts_tablet_manager.cc:1888] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.770833 11157 log.cc:1196] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/78bb6d55e8b44732b32da55f00071b21
I20240503 13:59:41.771183 7690 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.772220 7149 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:41.774444 9022 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.775614 10291 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.777951 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.778285 9283 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:41.778641 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.778807 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 14304180134241c792ec1f23ff89f94c with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:41.781414 11184 tablet_replica.cc:330] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.781620 11184 raft_consensus.cc:2229] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.781761 10291 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:41.781864 11184 raft_consensus.cc:2258] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.783885 11160 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.786887 7300 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.787016 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.789752 11184 ts_tablet_manager.cc:1875] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.791992 11184 ts_tablet_manager.cc:1888] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:41.792906 9025 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.733937 11200 ts_tablet_manager.cc:1888] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:41.794735 11200 log.cc:1196] T c74f828c28c64097ab4bf5ec66459d2f P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c74f828c28c64097ab4bf5ec66459d2f
I20240503 13:59:41.796039 7149 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.796626 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.797435 10298 raft_consensus.cc:1050] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:41.797819 10298 consensus_queue.cc:237] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.799530 9151 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.799804 11184 log.cc:1196] T 8aff688eaad64d149bdc1c8ab12f4ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8aff688eaad64d149bdc1c8ab12f4ca1
I20240503 13:59:41.799942 11160 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.800593 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 8aff688eaad64d149bdc1c8ab12f4ca1 (table rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]) successfully deleted
I20240503 13:59:41.804174 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet c74f828c28c64097ab4bf5ec66459d2f (table rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]) successfully deleted
I20240503 13:59:41.805441 11184 tablet_replica.cc:330] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.805584 7690 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.805831 11184 raft_consensus.cc:2229] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.806670 11184 raft_consensus.cc:2258] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.808367 11184 ts_tablet_manager.cc:1875] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.810002 11212 tablet_replica.cc:330] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: stopping tablet replica
I20240503 13:59:41.810361 11212 raft_consensus.cc:2229] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 13:59:41.810643 11212 pending_rounds.cc:62] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Trying to abort 1 pending ops.
I20240503 13:59:41.810827 11212 pending_rounds.cc:68] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023594624633163776 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "fca185e6c04648c5aa4f4ad51b0e1a39" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } } }
I20240503 13:59:41.811123 11212 raft_consensus.cc:2894] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 13:59:41.811245 11212 raft_consensus.cc:2866] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:41.811342 11212 raft_consensus.cc:2258] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 13:59:41.816682 11170 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.818883 7690 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.820421 11212 ts_tablet_manager.cc:1875] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.829880 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0b7095bdf3549fb83b3573a8e65deee with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.830185 7431 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.830417 11160 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.831864 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:41.831864 9283 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.833609 9419 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:41.833608 9025 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.799222 7562 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.833882 7300 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:41.834393 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:41.834427 10299 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.834744 10299 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:41.836432 9025 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:41.836774 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:41.836963 9283 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:41.837523 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:41.839287 11197 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.840041 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.840781 9025 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.841064 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33b7b43742e34431af0c16d320b35dd7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.842797 11158 raft_consensus.cc:1050] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.842612 7300 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.843215 11158 consensus_queue.cc:237] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.844904 11212 ts_tablet_manager.cc:1888] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:41.844803 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.845393 9022 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.845697 7149 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 9 to 10, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.845948 10201 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.846056 7149 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.846305 11212 log.cc:1196] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/fca185e6c04648c5aa4f4ad51b0e1a39
I20240503 13:59:41.846935 7145 catalog_manager.cc:4830] TS 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): tablet fca185e6c04648c5aa4f4ad51b0e1a39 (table rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]) successfully deleted
I20240503 13:59:41.847115 11184 ts_tablet_manager.cc:1888] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.847442 11184 log.cc:1196] T 491ee8399ef04ae7b93e3228fef05eb0 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/491ee8399ef04ae7b93e3228fef05eb0
I20240503 13:59:41.848064 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 491ee8399ef04ae7b93e3228fef05eb0 (table rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]) successfully deleted
I20240503 13:59:41.848095 11157 tablet_replica.cc:330] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:41.848362 11184 tablet_replica.cc:330] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.848362 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet 78bb6d55e8b44732b32da55f00071b21 (table rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]) successfully deleted
I20240503 13:59:41.848572 11184 raft_consensus.cc:2229] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.848793 11184 raft_consensus.cc:2258] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.848861 11157 raft_consensus.cc:2229] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.849045 11157 raft_consensus.cc:2258] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.849345 7300 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.849732 11157 ts_tablet_manager.cc:1875] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.849763 10201 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.855967 7692 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:41.856336 11184 ts_tablet_manager.cc:1875] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.857314 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet e0b7095bdf3549fb83b3573a8e65deee with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:41.857599 10201 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:41.858606 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b7b43742e34431af0c16d320b35dd7 with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:41.858937 11200 tablet_replica.cc:330] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:41.858940 7154 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 33b7b43742e34431af0c16d320b35dd7 with cas_config_opid_index 9: aborting the task: latest config opid_index 10; task opid_index 9
I20240503 13:59:41.859079 11200 raft_consensus.cc:2229] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.859300 11200 raft_consensus.cc:2258] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.860828 11200 ts_tablet_manager.cc:1875] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.861850 7431 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.862193 11213 tablet_replica.cc:330] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: stopping tablet replica
I20240503 13:59:41.862529 11213 raft_consensus.cc:2229] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.862807 11213 raft_consensus.cc:2258] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.865352 9283 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:41.865456 9419 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:41.865862 10298 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:41.866283 10298 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:41.868414 9025 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:41.868428 11200 ts_tablet_manager.cc:1888] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 13:59:41.868551 11200 log.cc:1196] T 33b7b43742e34431af0c16d320b35dd7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:41.869622 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:41.869818 10545 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.871039 11160 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.872382 9025 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.873083 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33b7b43742e34431af0c16d320b35dd7 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.874495 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.874564 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.877442 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet 33b7b43742e34431af0c16d320b35dd7 (table rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]) successfully deleted
I20240503 13:59:41.877442 7151 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 10 to 11, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.880529 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b7b43742e34431af0c16d320b35dd7 with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:41.886381 11184 ts_tablet_manager.cc:1888] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.887254 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:41.888098 10621 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.888383 10621 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:41.888579 10621 ts_tablet_manager.cc:1367] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.643s user 0.024s sys 0.000s
I20240503 13:59:41.889101 10621 raft_consensus.cc:348] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.889238 10621 raft_consensus.cc:729] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:41.889374 10621 consensus_queue.cc:260] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.892887 10554 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:41.893328 10554 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:41.893605 11213 ts_tablet_manager.cc:1875] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.894305 11184 log.cc:1196] T ff246d9b2c594282bf2e99d2fd2b617d P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff246d9b2c594282bf2e99d2fd2b617d
I20240503 13:59:41.894974 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet ff246d9b2c594282bf2e99d2fd2b617d (table rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]) successfully deleted
I20240503 13:59:41.895331 11184 tablet_replica.cc:330] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.895527 11184 raft_consensus.cc:2229] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.895815 11184 raft_consensus.cc:2258] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.896116 11213 ts_tablet_manager.cc:1888] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:41.896221 11213 log.cc:1196] T b23f9eeeb4af45b7b679b70514e247b8 P da390a1a407a4f30b217b3d60fba563f: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/b23f9eeeb4af45b7b679b70514e247b8
I20240503 13:59:41.896831 7145 catalog_manager.cc:4830] TS da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): tablet b23f9eeeb4af45b7b679b70514e247b8 (table rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]) successfully deleted
W20240503 13:59:41.897284 11131 consensus_queue.cc:1159] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e5c0b4b872d451dba60797739f83576" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240503 13:59:41.897950 11183 ts_tablet_manager.cc:1875] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.898887 10299 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.899502 7562 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.900214 11183 ts_tablet_manager.cc:1888] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:41.900274 11183 log.cc:1196] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0b7095bdf3549fb83b3573a8e65deee
I20240503 13:59:41.900689 11183 tablet_replica.cc:330] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:41.900816 11183 raft_consensus.cc:2229] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.900971 11183 raft_consensus.cc:2258] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.901156 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet e0b7095bdf3549fb83b3573a8e65deee (table rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]) successfully deleted
I20240503 13:59:41.901309 9151 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.902722 11183 ts_tablet_manager.cc:1875] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.893623 10554 ts_tablet_manager.cc:1367] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.499s user 0.023s sys 0.004s
I20240503 13:59:41.904840 7149 catalog_manager.cc:5484] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.905372 11184 ts_tablet_manager.cc:1875] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.906823 11183 ts_tablet_manager.cc:1888] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:41.906955 11183 log.cc:1196] T 14304180134241c792ec1f23ff89f94c P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/14304180134241c792ec1f23ff89f94c
I20240503 13:59:41.907616 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet 14304180134241c792ec1f23ff89f94c (table rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]) successfully deleted
I20240503 13:59:41.907718 11184 ts_tablet_manager.cc:1888] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:41.907788 11184 log.cc:1196] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b23f9eeeb4af45b7b679b70514e247b8
I20240503 13:59:41.908311 11184 tablet_replica.cc:330] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.908423 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet b23f9eeeb4af45b7b679b70514e247b8 (table rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]) successfully deleted
I20240503 13:59:41.908558 11184 raft_consensus.cc:2229] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.908773 11184 raft_consensus.cc:2258] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.913164 11184 ts_tablet_manager.cc:1875] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.914417 10197 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.917014 9022 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.917766 11197 raft_consensus.cc:1050] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:41.918263 11197 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.918269 7157 catalog_manager.cc:5484] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.920349 9022 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.921008 9283 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.921507 7299 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.921739 10299 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.922061 10299 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.922683 7692 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.924417 11198 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.924883 11157 ts_tablet_manager.cc:1888] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.925184 7562 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.925760 11157 log.cc:1196] T f73a5db0cae647d4b3375e6d42358307 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f73a5db0cae647d4b3375e6d42358307
I20240503 13:59:41.926357 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet f73a5db0cae647d4b3375e6d42358307 (table rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]) successfully deleted
I20240503 13:59:41.926663 11157 tablet_replica.cc:330] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:41.926857 11157 raft_consensus.cc:2229] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.927103 11157 raft_consensus.cc:2258] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.927228 11198 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.927655 11184 ts_tablet_manager.cc:1888] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:41.927736 11184 log.cc:1196] T c74f828c28c64097ab4bf5ec66459d2f P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c74f828c28c64097ab4bf5ec66459d2f
I20240503 13:59:41.928248 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet c74f828c28c64097ab4bf5ec66459d2f (table rebalance_test_table_2 [id=7f48fee88cf048bdbeea42628a36c726]) successfully deleted
I20240503 13:59:41.932047 7297 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.933439 7560 consensus_queue.cc:237] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:41.934701 11184 tablet_replica.cc:330] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:41.934861 11184 raft_consensus.cc:2229] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.935086 11184 raft_consensus.cc:2258] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.935331 9025 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.935876 11184 ts_tablet_manager.cc:1875] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.935843 9972 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.939065 7297 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.939842 9972 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.942512 7560 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.942698 7425 consensus_queue.cc:237] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:41.944011 7690 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.945250 7296 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.945472 9972 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.947677 9151 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.948081 7690 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.948107 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86634d81241c44cebcb602a6982f4244 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.948192 10299 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.948410 8919 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.948766 9025 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:41.949457 11158 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.949766 9670 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:41.949754 7155 catalog_manager.cc:5484] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.950825 10545 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:41.951045 10545 ts_tablet_manager.cc:1367] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 1.122s user 0.026s sys 0.000s
I20240503 13:59:41.951391 8919 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.951542 9670 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.951867 11157 ts_tablet_manager.cc:1875] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.953022 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dffd8718851475099a6696e4a2b8901 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:41.953863 9151 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.954113 9025 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.955240 11157 ts_tablet_manager.cc:1888] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:41.955308 11157 log.cc:1196] T e0b7095bdf3549fb83b3573a8e65deee P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e0b7095bdf3549fb83b3573a8e65deee
I20240503 13:59:41.955663 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 86634d81241c44cebcb602a6982f4244 with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:41.955895 11200 tablet_replica.cc:330] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:41.956033 11200 raft_consensus.cc:2229] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.956195 11200 raft_consensus.cc:2258] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.956485 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet e0b7095bdf3549fb83b3573a8e65deee (table rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]) successfully deleted
I20240503 13:59:41.956799 7154 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.957298 11200 ts_tablet_manager.cc:1875] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.957441 7157 catalog_manager.cc:5484] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:41.959268 9024 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.959432 7300 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:41.959610 7690 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:41.961932 11200 ts_tablet_manager.cc:1888] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:41.962035 11200 log.cc:1196] T 86634d81241c44cebcb602a6982f4244 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/86634d81241c44cebcb602a6982f4244
W20240503 13:59:41.962951 7498 consensus_peers.cc:485] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 110ac05b2e8c4594a03225ab2c1ad56d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:41.963291 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dffd8718851475099a6696e4a2b8901 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:41.964674 7297 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:41.965350 11198 raft_consensus.cc:1050] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:41.965845 11198 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:41.967528 7560 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.967948 11197 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.973459 9024 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.974099 8919 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.975328 10554 raft_consensus.cc:348] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.975468 10554 raft_consensus.cc:729] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:41.975561 10554 consensus_queue.cc:260] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.975872 10554 ts_tablet_manager.cc:1398] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.072s user 0.001s sys 0.000s
I20240503 13:59:41.975942 10621 ts_tablet_manager.cc:1398] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.087s user 0.003s sys 0.000s
I20240503 13:59:41.977006 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-faa9fb136138439b8f694ac95dbfa2e7 received from {username='slave'} at 127.0.99.136:33557
I20240503 13:59:41.977088 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-faa9fb136138439b8f694ac95dbfa2e7 on tablet faa9fb136138439b8f694ac95dbfa2e7 with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:41.977494 11157 tablet_replica.cc:330] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:41.977466 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-3622884b9b314ccca24422d8fa05360a received from {username='slave'} at 127.0.99.136:56397
I20240503 13:59:41.977537 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-3622884b9b314ccca24422d8fa05360a on tablet 3622884b9b314ccca24422d8fa05360a with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:41.977622 11157 raft_consensus.cc:2229] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.977800 11157 raft_consensus.cc:2258] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.979940 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet 86634d81241c44cebcb602a6982f4244 (table rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]) successfully deleted
I20240503 13:59:41.980530 11157 ts_tablet_manager.cc:1875] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.980595 7692 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.981043 8919 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.982714 10545 raft_consensus.cc:348] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.982923 10545 raft_consensus.cc:729] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:41.983072 10545 consensus_queue.cc:260] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:41.984045 10545 ts_tablet_manager.cc:1398] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20240503 13:59:41.984547 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-a4b0fd771f844fcab38aec069b9abd00 received from {username='slave'} at 127.0.99.136:56397
I20240503 13:59:41.984633 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-a4b0fd771f844fcab38aec069b9abd00 on tablet a4b0fd771f844fcab38aec069b9abd00 with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:41.982933 9419 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:41.986651 8919 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:41.988690 11184 ts_tablet_manager.cc:1888] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 13:59:41.988848 11184 log.cc:1196] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:41.989451 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 33b7b43742e34431af0c16d320b35dd7 (table rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]) successfully deleted
I20240503 13:59:41.991840 8919 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.995747 7560 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.997840 9418 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.998034 10349 raft_consensus.cc:1013] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: attempt to promote peer c5c3042421284f028fa512b7876da405: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 13:59:41.999204 7690 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:41.998554 7153 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.002955 11198 raft_consensus.cc:1050] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:42.003440 11198 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:42.005072 9418 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:42.005365 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:42.005472 11037 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:42.005682 9151 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.005772 11037 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:42.005889 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.007404 7560 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.007643 11197 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.008208 11197 raft_consensus.cc:1050] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:42.008589 11197 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
W20240503 13:59:42.009894 7143 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet c1a85f321bfa4faf9b48cb319221fb94 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.010131 11157 ts_tablet_manager.cc:1888] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:42.010115 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.010233 11157 log.cc:1196] T 4dffd8718851475099a6696e4a2b8901 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dffd8718851475099a6696e4a2b8901
I20240503 13:59:42.011085 9025 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.012543 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet 4dffd8718851475099a6696e4a2b8901 (table rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]) successfully deleted
I20240503 13:59:42.013058 10299 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.005964 11037 ts_tablet_manager.cc:1367] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.833s user 0.024s sys 0.001s
I20240503 13:59:42.014182 11037 raft_consensus.cc:348] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:42.014384 11037 raft_consensus.cc:729] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:42.014520 11037 consensus_queue.cc:260] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:42.014410 9025 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.014894 11037 ts_tablet_manager.cc:1398] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:42.016507 7560 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.016954 10299 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.019327 10299 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.022192 7299 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.024837 7562 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.027593 9283 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.027714 7155 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.029383 7692 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.030079 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-faa9fb136138439b8f694ac95dbfa2e7 received from {username='slave'} at 127.0.99.134:45115
I20240503 13:59:42.030162 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-faa9fb136138439b8f694ac95dbfa2e7 on tablet faa9fb136138439b8f694ac95dbfa2e7 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:42.030087 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.030012 9022 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.031098 9419 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.031674 10299 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.031934 7425 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:42.035642 7562 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.035691 9022 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.036142 7299 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.036846 10298 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.038277 8919 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.039336 10628 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:42.039495 10299 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.041288 10299 raft_consensus.cc:1013] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: attempt to promote peer da390a1a407a4f30b217b3d60fba563f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 13:59:42.044416 7297 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:42.045572 10628 tablet_bootstrap.cc:492] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:42.046205 7693 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:42.046547 10628 ts_tablet_manager.cc:1367] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.966s user 0.028s sys 0.000s
I20240503 13:59:42.046772 10752 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:42.047056 10628 raft_consensus.cc:348] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:42.047184 10628 raft_consensus.cc:729] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
W20240503 13:59:42.047178 7236 consensus_peers.cc:485] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23cf5703ccef47ea927342b9c29f820a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:42.047293 10628 consensus_queue.cc:260] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:42.047427 7692 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.047587 10628 ts_tablet_manager.cc:1398] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:42.048391 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-2212cf72b6ee4ee5b086455a75482e3b received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:42.048481 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-2212cf72b6ee4ee5b086455a75482e3b on tablet 2212cf72b6ee4ee5b086455a75482e3b with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:42.049530 7561 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:42.049628 8919 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.050503 10752 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:42.053128 7558 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:42.053867 8919 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.054611 9151 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.054625 7299 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:42.054793 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade40ddaf0304b1a9e94db6630175c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:42.054976 10201 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.055352 11156 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.056569 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e2cabe02dbf4580b53875aaa09aaffa with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.056746 7297 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.057368 11156 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.057829 9022 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.058300 7558 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.058495 7151 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.059649 7299 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.060689 9151 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.063192 7154 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.066411 10752 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.066955 10349 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.067713 7425 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:42.068885 7561 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.069353 9151 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.069535 9022 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.069921 11160 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.072633 10299 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.073119 7149 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.074219 9267 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2cabe02dbf4580b53875aaa09aaffa with delete_type TABLET_DATA_TOMBSTONED (TS c5c3042421284f028fa512b7876da405 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45882
I20240503 13:59:42.074657 7558 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.075018 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e2cabe02dbf4580b53875aaa09aaffa with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.075194 9022 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.076331 7561 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.076217 7691 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.072216 9419 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.079066 7155 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.079152 11197 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.080713 7561 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:42.084169 7425 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:42.084271 11197 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.084355 7151 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.086134 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2cabe02dbf4580b53875aaa09aaffa with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:42.086499 11184 tablet_replica.cc:330] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:42.086704 11184 raft_consensus.cc:2229] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.086971 11184 raft_consensus.cc:2258] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.086978 9022 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.087814 9151 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.088104 11184 ts_tablet_manager.cc:1875] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.088354 11160 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.089965 9417 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.089798 7154 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.090363 7692 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.090688 7691 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.092281 11160 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.094321 7561 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.094789 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.096383 9418 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.096804 11075 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:42.097084 11075 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:42.097268 11075 ts_tablet_manager.cc:1367] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.972s user 0.023s sys 0.004s
I20240503 13:59:42.097998 8919 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.098356 7425 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.100242 9417 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.100677 11160 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.100955 7560 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.101066 9022 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.100668 11197 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.101596 8919 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.102767 10299 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.102844 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2212cf72b6ee4ee5b086455a75482e3b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.103945 11198 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.104357 9151 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.106361 9418 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.107381 11160 raft_consensus.cc:1050] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:42.107448 7690 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:42.107726 11160 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:42.108206 9022 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.109661 9418 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.111189 11075 raft_consensus.cc:348] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:42.111482 11075 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:42.111660 11075 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:42.111884 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c1a85f321bfa4faf9b48cb319221fb94 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.112165 11075 ts_tablet_manager.cc:1398] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:59:42.112287 7560 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.112706 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-14304180134241c792ec1f23ff89f94c received from {username='slave'} at 127.0.99.134:38323
I20240503 13:59:42.112795 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-14304180134241c792ec1f23ff89f94c on tablet 14304180134241c792ec1f23ff89f94c with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:42.112749 7561 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.113566 11269 tablet_replica.cc:330] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: stopping tablet replica
I20240503 13:59:42.113948 7149 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.114476 9151 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.114518 7562 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.114840 9419 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.114888 11197 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.115316 11269 raft_consensus.cc:2229] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 13:59:42.115538 11197 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.115746 11269 raft_consensus.cc:2258] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 13:59:42.116112 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.123286 11269 ts_tablet_manager.cc:1875] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.124948 7143 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet c1a85f321bfa4faf9b48cb319221fb94 with cas_config_opid_index 6: aborting the task: latest config opid_index 8; task opid_index 6
I20240503 13:59:42.128096 10299 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.130108 7562 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.130326 9151 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.131816 9152 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.131845 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a85f321bfa4faf9b48cb319221fb94 with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:42.131757 7691 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.132426 11200 tablet_replica.cc:330] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:42.132571 11200 raft_consensus.cc:2229] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.132761 11200 raft_consensus.cc:2258] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.133392 7692 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.133361 10385 consensus_queue.cc:1037] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.134182 9152 raft_consensus.cc:1203] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:42.134428 11200 ts_tablet_manager.cc:1875] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.135623 9151 raft_consensus.cc:858] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
W20240503 13:59:42.135694 9151 raft_consensus.cc:1565] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:42.136091 9151 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:42.136680 11184 ts_tablet_manager.cc:1888] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:42.136840 11184 log.cc:1196] T 4e2cabe02dbf4580b53875aaa09aaffa P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4e2cabe02dbf4580b53875aaa09aaffa
I20240503 13:59:42.137490 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 4e2cabe02dbf4580b53875aaa09aaffa (table rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]) successfully deleted
I20240503 13:59:42.138769 11200 ts_tablet_manager.cc:1888] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:42.138834 11200 log.cc:1196] T c1a85f321bfa4faf9b48cb319221fb94 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c1a85f321bfa4faf9b48cb319221fb94
I20240503 13:59:42.139618 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet c1a85f321bfa4faf9b48cb319221fb94 (table rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]) successfully deleted
I20240503 13:59:42.139772 7154 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.142771 11265 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.143335 9418 raft_consensus.cc:1203] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 13:59:42.144096 7692 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.147069 9267 tablet_service.cc:1514] Processing DeleteTablet for tablet 2212cf72b6ee4ee5b086455a75482e3b with delete_type TABLET_DATA_TOMBSTONED (TS c5c3042421284f028fa512b7876da405 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45882
I20240503 13:59:42.148030 7425 consensus_queue.cc:237] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.149523 9419 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.150025 7157 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 8 to 9, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.151860 9418 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.152865 9022 raft_consensus.cc:1261] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.153894 11265 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.154059 11160 consensus_queue.cc:1035] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.156607 7425 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:42.158190 9152 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:42.158727 11160 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:42.159498 10298 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.160056 9418 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:42.161376 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c1a85f321bfa4faf9b48cb319221fb94 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.161567 7692 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:42.162719 10298 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:42.164424 7155 catalog_manager.cc:5484] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.167212 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a85f321bfa4faf9b48cb319221fb94 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:42.167423 11157 tablet_replica.cc:330] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:42.167553 11157 raft_consensus.cc:2229] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.167575 10298 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:42.167738 11157 pending_rounds.cc:62] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Trying to abort 1 pending ops.
I20240503 13:59:42.167694 9418 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.167804 11157 pending_rounds.cc:68] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023594627472998400 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c1a85f321bfa4faf9b48cb319221fb94" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } } }
I20240503 13:59:42.167986 11157 raft_consensus.cc:2894] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 13:59:42.168047 11157 raft_consensus.cc:2866] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:42.168094 11157 raft_consensus.cc:2258] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.169342 11157 ts_tablet_manager.cc:1875] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.169283 11197 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.169521 9022 raft_consensus.cc:2932] T c1a85f321bfa4faf9b48cb319221fb94 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.171022 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2212cf72b6ee4ee5b086455a75482e3b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.172010 11157 ts_tablet_manager.cc:1888] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:42.172083 11157 log.cc:1196] T c1a85f321bfa4faf9b48cb319221fb94 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c1a85f321bfa4faf9b48cb319221fb94
I20240503 13:59:42.172449 9417 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.172454 9152 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.172922 11269 ts_tablet_manager.cc:1888] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:42.173650 11269 log.cc:1196] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4e2cabe02dbf4580b53875aaa09aaffa
I20240503 13:59:42.174387 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet c1a85f321bfa4faf9b48cb319221fb94 (table rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]) successfully deleted
I20240503 13:59:42.175832 7145 catalog_manager.cc:4830] TS c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): tablet 4e2cabe02dbf4580b53875aaa09aaffa (table rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]) successfully deleted
I20240503 13:59:42.176129 7153 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 9 to 10, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.176568 11269 tablet_replica.cc:330] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: stopping tablet replica
I20240503 13:59:42.177552 7691 consensus_queue.cc:237] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:42.177649 7692 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.179432 7299 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:42.180186 7427 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:42.180975 10291 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:59:42.182191 7629 consensus_peers.cc:485] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93d454886dfa40018a4403a3d74b3de2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:42.182396 11269 raft_consensus.cc:2229] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 13:59:42.182531 11269 raft_consensus.cc:2258] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 13:59:42.183161 11269 ts_tablet_manager.cc:1875] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.184978 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 2212cf72b6ee4ee5b086455a75482e3b with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:42.186074 11157 tablet_replica.cc:330] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:42.186267 11157 raft_consensus.cc:2229] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.186507 11157 raft_consensus.cc:2258] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.189277 7561 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:42.189339 11269 ts_tablet_manager.cc:1888] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:42.189419 11269 log.cc:1196] T 2212cf72b6ee4ee5b086455a75482e3b P c5c3042421284f028fa512b7876da405: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2212cf72b6ee4ee5b086455a75482e3b
I20240503 13:59:42.189728 11157 ts_tablet_manager.cc:1875] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.189779 9152 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:42.190016 7145 catalog_manager.cc:4830] TS c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): tablet 2212cf72b6ee4ee5b086455a75482e3b (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:42.190387 10298 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:42.190639 9418 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:42.191088 10299 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:42.193138 11265 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.193579 9152 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.193696 7427 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } }
I20240503 13:59:42.195621 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2212cf72b6ee4ee5b086455a75482e3b with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.195662 9283 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.196197 10299 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.196398 10291 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:42.196738 7149 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.197830 11157 ts_tablet_manager.cc:1888] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 13:59:42.198000 11157 log.cc:1196] T 2212cf72b6ee4ee5b086455a75482e3b P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2212cf72b6ee4ee5b086455a75482e3b
I20240503 13:59:42.197942 10299 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:42.198684 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet 2212cf72b6ee4ee5b086455a75482e3b (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:42.199645 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fca185e6c04648c5aa4f4ad51b0e1a39 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.200126 7690 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.200623 9283 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:42.200822 8919 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.201573 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 2212cf72b6ee4ee5b086455a75482e3b with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:42.202414 7690 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:42.202600 9417 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.206238 7149 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 7 to 8, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } }
I20240503 13:59:42.207880 11200 tablet_replica.cc:330] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:42.208048 11200 raft_consensus.cc:2229] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.208271 11200 raft_consensus.cc:2258] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.209264 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet fca185e6c04648c5aa4f4ad51b0e1a39 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:42.209542 11157 tablet_replica.cc:330] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:42.209718 11157 raft_consensus.cc:2229] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.210283 11157 raft_consensus.cc:2258] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.211220 11157 ts_tablet_manager.cc:1875] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.213596 11157 ts_tablet_manager.cc:1888] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:42.213686 11157 log.cc:1196] T fca185e6c04648c5aa4f4ad51b0e1a39 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fca185e6c04648c5aa4f4ad51b0e1a39
I20240503 13:59:42.214421 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet fca185e6c04648c5aa4f4ad51b0e1a39 (table rebalance_test_table_1 [id=c4211301acdd465c9d7742a40279a30c]) successfully deleted
I20240503 13:59:42.215013 11200 ts_tablet_manager.cc:1875] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.217515 10291 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.219689 7299 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.221320 7690 consensus_queue.cc:237] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:42.223799 7427 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:42.224030 7153 catalog_manager.cc:5484] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.224323 7299 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:42.224905 10291 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:42.225101 10082 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:42.227676 9418 raft_consensus.cc:1203] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:42.230393 7561 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:42.235252 11170 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:42.237250 11200 ts_tablet_manager.cc:1888] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 13:59:42.237365 11200 log.cc:1196] T 2212cf72b6ee4ee5b086455a75482e3b P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2212cf72b6ee4ee5b086455a75482e3b
I20240503 13:59:42.238075 7299 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:42.241950 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet 2212cf72b6ee4ee5b086455a75482e3b (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:42.243126 7149 catalog_manager.cc:5484] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:42.253724 10385 raft_consensus.cc:1050] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:42.253736 7425 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.254995 9151 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:42.256067 9418 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.256525 11265 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.256798 10385 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:42.257063 9283 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.257755 11265 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.259099 7690 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.260277 7561 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.260778 11265 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.260802 10298 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.261418 9283 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.261792 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.263051 9418 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.262960 9417 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.263238 7427 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:42.263413 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 902cf3db3581465c83170c4b13b3bffb with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.263455 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.264369 7561 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.265478 10276 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.265807 7157 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 7 to 8, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.269028 9418 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.269203 7154 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.271443 7690 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.273774 9152 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.274734 10142 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.276018 9152 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.279659 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 902cf3db3581465c83170c4b13b3bffb with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:42.279913 11184 tablet_replica.cc:330] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:42.280066 11184 raft_consensus.cc:2229] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.280083 7427 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.280237 11184 raft_consensus.cc:2258] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.281241 11184 ts_tablet_manager.cc:1875] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.281824 9418 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.281966 9283 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.282496 10299 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.282814 10299 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.283816 11184 ts_tablet_manager.cc:1888] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:42.283886 11184 log.cc:1196] T 902cf3db3581465c83170c4b13b3bffb P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/902cf3db3581465c83170c4b13b3bffb
I20240503 13:59:42.284377 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 902cf3db3581465c83170c4b13b3bffb (table rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]) successfully deleted
I20240503 13:59:42.284924 11265 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.286847 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 902cf3db3581465c83170c4b13b3bffb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.287987 7153 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.288165 9283 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.287681 9418 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.290895 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 902cf3db3581465c83170c4b13b3bffb with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:42.291129 11157 tablet_replica.cc:330] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:42.291270 11157 raft_consensus.cc:2229] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.291512 11157 raft_consensus.cc:2258] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.292450 11157 ts_tablet_manager.cc:1875] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.294875 11157 ts_tablet_manager.cc:1888] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:42.294960 11157 log.cc:1196] T 902cf3db3581465c83170c4b13b3bffb P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/902cf3db3581465c83170c4b13b3bffb
I20240503 13:59:42.295615 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet 902cf3db3581465c83170c4b13b3bffb (table rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]) successfully deleted
W20240503 13:59:42.309351 7629 consensus_peers.cc:485] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b04204ad281452794d724fb4d7212f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:42.330602 9418 raft_consensus.cc:1203] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:42.476598 11438 ts_tablet_manager.cc:902] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:42.477195 11438 tablet_copy_client.cc:308] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:42.477831 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 23cf5703ccef47ea927342b9c29f820a from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:58407)
I20240503 13:59:42.478010 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 23cf5703ccef47ea927342b9c29f820a from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:58407: session id = 476f53c7e40c4a0b86ffbd818aea818d-23cf5703ccef47ea927342b9c29f820a
I20240503 13:59:42.479027 7320 tablet_copy_source_session.cc:205] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:42.480468 11439 ts_tablet_manager.cc:902] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:42.481042 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23cf5703ccef47ea927342b9c29f820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.481150 11439 tablet_copy_client.cc:308] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:42.481784 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet ade40ddaf0304b1a9e94db6630175c0e from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:50181)
I20240503 13:59:42.481900 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet ade40ddaf0304b1a9e94db6630175c0e from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:50181: session id = 476f53c7e40c4a0b86ffbd818aea818d-ade40ddaf0304b1a9e94db6630175c0e
I20240503 13:59:42.482889 7582 tablet_copy_source_session.cc:205] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:42.484170 11439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade40ddaf0304b1a9e94db6630175c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.492941 11439 tablet_copy_client.cc:783] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:42.493244 11439 tablet_copy_client.cc:647] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:42.494359 11438 tablet_copy_client.cc:783] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:42.494598 11438 tablet_copy_client.cc:647] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:42.494699 11439 tablet_copy_client.cc:523] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:42.496385 11439 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:42.497599 11438 tablet_copy_client.cc:523] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:42.501169 11438 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:42.527690 9972 raft_consensus.cc:1050] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:42.528302 9972 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.531788 7427 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.532239 10197 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.532557 7690 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.533965 10197 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.534822 9152 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.536182 11509 ts_tablet_manager.cc:902] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:42.536231 9418 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.537158 11509 tablet_copy_client.cc:308] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:42.538204 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 110ac05b2e8c4594a03225ab2c1ad56d from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:33557)
I20240503 13:59:42.538326 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 110ac05b2e8c4594a03225ab2c1ad56d from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:33557: session id = da390a1a407a4f30b217b3d60fba563f-110ac05b2e8c4594a03225ab2c1ad56d
I20240503 13:59:42.539323 7582 tablet_copy_source_session.cc:205] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:42.539693 9670 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.540068 10201 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.542840 11509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110ac05b2e8c4594a03225ab2c1ad56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.546941 11509 tablet_copy_client.cc:783] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:42.547192 11509 tablet_copy_client.cc:647] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:42.555011 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.555469 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.568598 10197 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.569077 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.571287 9418 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.572136 7690 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.575457 7151 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.578027 10201 raft_consensus.cc:1050] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:42.578536 10201 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.578848 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.580987 7690 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.582087 9152 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.583673 9418 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.584306 9972 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.584612 9972 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:42.584853 9972 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.586668 9152 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.592628 7425 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:42.593204 10197 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.594246 7425 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.597061 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.597122 11156 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.600639 7690 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.602870 7425 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.604513 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.610962 7150 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.615716 9152 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.618480 9418 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.624876 7562 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.627486 7690 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.627949 9972 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.628643 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.631549 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.644188 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.646417 10752 raft_consensus.cc:1050] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:42.646713 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.646867 10752 consensus_queue.cc:237] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.648993 7425 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:42.649188 9418 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:42.649435 10142 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.649688 10385 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.650296 7690 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:42.650763 10385 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.652668 9417 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.652835 9152 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:42.653215 11156 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.653508 11156 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:42.656553 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.656773 11156 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.657708 7142 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet faa9fb136138439b8f694ac95dbfa2e7 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240503 13:59:42.659632 7692 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.659984 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet faa9fb136138439b8f694ac95dbfa2e7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.660914 7153 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.662608 9152 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.664582 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet faa9fb136138439b8f694ac95dbfa2e7 with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:42.667934 9417 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.671196 7562 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:42.672492 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.672685 7692 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.674408 10197 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.675503 9151 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:42.677421 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.677551 10197 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:42.680025 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.681469 10201 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.684163 7692 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.685071 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.684994 7150 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.687817 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.688928 9152 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:42.691406 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet faa9fb136138439b8f694ac95dbfa2e7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.692996 9398 tablet_service.cc:1514] Processing DeleteTablet for tablet faa9fb136138439b8f694ac95dbfa2e7 with delete_type TABLET_DATA_TOMBSTONED (TS da390a1a407a4f30b217b3d60fba563f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50322
I20240503 13:59:42.695060 11540 ts_tablet_manager.cc:902] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:42.695504 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.696230 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.705367 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.707729 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.719305 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.723135 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.723873 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.752056 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.753207 11545 tablet_replica.cc:330] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: stopping tablet replica
I20240503 13:59:42.753449 11545 raft_consensus.cc:2229] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.753741 11545 raft_consensus.cc:2258] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.762619 11545 ts_tablet_manager.cc:1875] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.762670 11540 tablet_copy_client.cc:308] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:42.763248 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 93d454886dfa40018a4403a3d74b3de2 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:43367)
I20240503 13:59:42.763368 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 93d454886dfa40018a4403a3d74b3de2 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:43367: session id = c5c3042421284f028fa512b7876da405-93d454886dfa40018a4403a3d74b3de2
I20240503 13:59:42.763422 11551 ts_tablet_manager.cc:902] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:42.763442 10142 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.763900 11551 tablet_copy_client.cc:308] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:42.764343 7710 tablet_copy_source_session.cc:205] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:42.767169 11545 ts_tablet_manager.cc:1888] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:42.767663 11540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d454886dfa40018a4403a3d74b3de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.767691 11546 tablet_replica.cc:330] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: stopping tablet replica
I20240503 13:59:42.767613 7153 catalog_manager.cc:5484] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.767864 11546 raft_consensus.cc:2229] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.768052 11546 raft_consensus.cc:2258] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.770324 7690 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.771921 11540 tablet_copy_client.cc:783] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:42.772090 11540 tablet_copy_client.cc:647] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:42.774504 7299 consensus_queue.cc:237] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.776578 9417 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.777465 10276 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.778741 10276 raft_consensus.cc:1050] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:42.778759 9417 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.779139 10276 consensus_queue.cc:237] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.780802 7562 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:42.781797 10385 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.782653 9417 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:42.783000 10385 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.784865 7690 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:42.785192 11540 tablet_copy_client.cc:523] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:42.785629 10142 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:42.786371 11546 ts_tablet_manager.cc:1875] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.787426 11540 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:42.789209 11546 ts_tablet_manager.cc:1888] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 13:59:42.799958 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.801573 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.806205 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:42.809955 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 9b04204ad281452794d724fb4d7212f8 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:58515)
I20240503 13:59:42.810097 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 9b04204ad281452794d724fb4d7212f8 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:58515: session id = 708734e6729441fcb2c5952fa47771d9-9b04204ad281452794d724fb4d7212f8
I20240503 13:59:42.810148 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.811120 7710 tablet_copy_source_session.cc:205] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:42.812180 11551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b04204ad281452794d724fb4d7212f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:42.815410 11551 tablet_copy_client.cc:783] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:42.815670 11551 tablet_copy_client.cc:647] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:42.817247 11551 tablet_copy_client.cc:523] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:42.819411 11551 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:42.821616 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.834136 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.835312 10385 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.839087 7153 catalog_manager.cc:5484] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.841733 7562 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.842458 7299 consensus_queue.cc:237] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:42.850673 7562 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.850940 10142 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.851279 10752 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.852854 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4b0fd771f844fcab38aec069b9abd00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.858549 7151 catalog_manager.cc:5484] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:42.861879 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b0fd771f844fcab38aec069b9abd00 with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:42.768967 7425 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.795028 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.869463 11545 log.cc:1196] T faa9fb136138439b8f694ac95dbfa2e7 P 1e5c0b4b872d451dba60797739f83576: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/faa9fb136138439b8f694ac95dbfa2e7
I20240503 13:59:42.871043 10385 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.871803 7427 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.872812 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3622884b9b314ccca24422d8fa05360a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:42.873895 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet faa9fb136138439b8f694ac95dbfa2e7 (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:42.836241 9417 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.874708 10752 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.838430 11509 tablet_copy_client.cc:523] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:42.875329 7151 catalog_manager.cc:5484] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:42.875639 7562 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.876716 11509 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:42.853742 9419 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.881273 7427 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.882125 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 3622884b9b314ccca24422d8fa05360a with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:42.885632 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:42.890856 9418 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:42.891558 10385 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:42.892581 9418 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:42.904552 11546 log.cc:1196] T faa9fb136138439b8f694ac95dbfa2e7 P da390a1a407a4f30b217b3d60fba563f: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/faa9fb136138439b8f694ac95dbfa2e7
I20240503 13:59:42.905263 11438 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:42.905259 7145 catalog_manager.cc:4830] TS da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): tablet faa9fb136138439b8f694ac95dbfa2e7 (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:42.905827 11438 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:42.906229 11438 ts_tablet_manager.cc:1367] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.405s user 0.028s sys 0.000s
I20240503 13:59:42.918896 11572 tablet_replica.cc:330] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:42.931301 11572 raft_consensus.cc:2229] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:42.931679 11572 raft_consensus.cc:2258] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:42.933943 11438 raft_consensus.cc:348] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:42.934264 11438 raft_consensus.cc:729] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:42.934806 11438 consensus_queue.cc:260] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:42.935290 11438 ts_tablet_manager.cc:1398] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:42.935932 11572 ts_tablet_manager.cc:1875] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:42.918396 11439 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:42.938120 11439 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:42.938413 11439 ts_tablet_manager.cc:1367] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.442s user 0.025s sys 0.004s
I20240503 13:59:42.938789 11572 ts_tablet_manager.cc:1888] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:42.939000 11572 log.cc:1196] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a4b0fd771f844fcab38aec069b9abd00
I20240503 13:59:42.938951 11439 raft_consensus.cc:348] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:42.939169 11439 raft_consensus.cc:729] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:42.939325 11439 consensus_queue.cc:260] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:42.939597 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet a4b0fd771f844fcab38aec069b9abd00 (table rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]) successfully deleted
I20240503 13:59:42.939775 11439 ts_tablet_manager.cc:1398] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:42.950148 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-ade40ddaf0304b1a9e94db6630175c0e received from {username='slave'} at 127.0.99.133:50181
I20240503 13:59:42.950268 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-ade40ddaf0304b1a9e94db6630175c0e on tablet ade40ddaf0304b1a9e94db6630175c0e with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:42.959910 9024 raft_consensus.cc:1203] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:42.962643 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-23cf5703ccef47ea927342b9c29f820a received from {username='slave'} at 127.0.99.133:58407
I20240503 13:59:42.962751 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-23cf5703ccef47ea927342b9c29f820a on tablet 23cf5703ccef47ea927342b9c29f820a with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:43.006184 11572 tablet_replica.cc:330] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:43.006476 11572 raft_consensus.cc:2229] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.006700 11572 pending_rounds.cc:62] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Trying to abort 1 pending ops.
I20240503 13:59:43.006824 11572 pending_rounds.cc:68] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023594630262255616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3622884b9b314ccca24422d8fa05360a" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } } }
I20240503 13:59:43.007146 11572 raft_consensus.cc:2894] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 13:59:43.007263 11572 raft_consensus.cc:2866] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 13:59:43.007361 11572 raft_consensus.cc:2258] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.011540 11540 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.012061 11540 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:43.012306 11540 ts_tablet_manager.cc:1367] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.225s user 0.023s sys 0.005s
I20240503 13:59:43.012835 11540 raft_consensus.cc:348] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.013056 11540 raft_consensus.cc:729] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:43.013216 11540 consensus_queue.cc:260] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.013646 11540 ts_tablet_manager.cc:1398] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.014278 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session c5c3042421284f028fa512b7876da405-93d454886dfa40018a4403a3d74b3de2 received from {username='slave'} at 127.0.99.135:43367
I20240503 13:59:43.014358 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session c5c3042421284f028fa512b7876da405-93d454886dfa40018a4403a3d74b3de2 on tablet 93d454886dfa40018a4403a3d74b3de2 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:43.014528 11509 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.014833 11509 tablet_bootstrap.cc:492] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:43.015049 11509 ts_tablet_manager.cc:1367] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.138s user 0.029s sys 0.000s
I20240503 13:59:43.015547 11509 raft_consensus.cc:348] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:43.015676 11509 raft_consensus.cc:729] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:43.015780 11509 consensus_queue.cc:260] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:43.016062 11509 ts_tablet_manager.cc:1398] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:43.017433 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-110ac05b2e8c4594a03225ab2c1ad56d received from {username='slave'} at 127.0.99.136:33557
I20240503 13:59:43.017510 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-110ac05b2e8c4594a03225ab2c1ad56d on tablet 110ac05b2e8c4594a03225ab2c1ad56d with peer da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:43.021199 11551 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.021627 11551 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:43.021904 11551 ts_tablet_manager.cc:1367] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.203s user 0.029s sys 0.000s
I20240503 13:59:43.022661 11551 raft_consensus.cc:348] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.022805 11551 raft_consensus.cc:729] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:43.022931 11551 consensus_queue.cc:260] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.023203 11551 ts_tablet_manager.cc:1398] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.023770 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-9b04204ad281452794d724fb4d7212f8 received from {username='slave'} at 127.0.99.134:58515
I20240503 13:59:43.023851 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-9b04204ad281452794d724fb4d7212f8 on tablet 9b04204ad281452794d724fb4d7212f8 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:43.031766 7427 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } }
I20240503 13:59:43.033402 11572 ts_tablet_manager.cc:1875] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.033438 9283 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:43.033587 9024 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:43.033716 9418 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:43.034382 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:43.034569 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:43.034878 10299 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:43.036337 11572 ts_tablet_manager.cc:1888] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:43.036470 11572 log.cc:1196] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3622884b9b314ccca24422d8fa05360a
I20240503 13:59:43.037072 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet 3622884b9b314ccca24422d8fa05360a (table rebalance_test_table_0 [id=5b979a355c5c46acac1c80a82e9b5d33]) successfully deleted
I20240503 13:59:43.041844 10299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } }
I20240503 13:59:43.045889 7150 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 11 to 12. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } }
I20240503 13:59:43.049225 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } }
I20240503 13:59:43.049455 9024 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } }
I20240503 13:59:43.054591 7427 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:43.056769 9283 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240503 13:59:43.057129 7364 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): Couldn't send request to peer e1469ca6a5f945ec89ee1eb8df023b96. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.057245 10299 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:43.057832 9024 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 13:59:43.059221 10299 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:43.063781 9024 raft_consensus.cc:1203] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:43.068472 9285 raft_consensus.cc:1203] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:43.075376 7690 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.077354 9417 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 13:59:43.077804 7427 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.077365 9419 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.078516 10082 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.078913 11197 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:43.079268 10299 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
W20240503 13:59:43.079816 7629 consensus_peers.cc:485] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff246d9b2c594282bf2e99d2fd2b617d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.079946 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.080488 11608 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.082135 7143 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 33b7b43742e34431af0c16d320b35dd7 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:59:43.082926 7149 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 12 to 13, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.083015 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } }
I20240503 13:59:43.084470 9419 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.085242 9024 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.087280 7299 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:43.088953 7690 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.089354 9152 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.089735 10385 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.089931 10142 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.091614 9417 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.092034 10385 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.093400 10385 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.096321 7690 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.097712 7152 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:43.099934 9418 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.101759 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.102700 7299 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.103385 10082 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.106276 9151 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.106498 9152 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.108894 9418 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.109580 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.109639 7690 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.110244 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.112586 10385 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.113440 10385 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.117707 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 13:59:43.118072 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14304180134241c792ec1f23ff89f94c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.118754 7152 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.119825 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.123294 7427 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.125560 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.130501 7562 consensus_queue.cc:237] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.132781 7299 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:43.133193 10197 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:43.134840 7692 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:43.135128 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.135672 10197 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:43.138796 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.144080 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.146731 10276 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.148857 7142 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 14304180134241c792ec1f23ff89f94c with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:59:43.151260 9152 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.151118 7154 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 9 to 10, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.154436 7690 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.157748 9419 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.158468 7299 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.161552 9152 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 13:59:43.161671 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14304180134241c792ec1f23ff89f94c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.162045 10752 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:43.163810 10752 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.165589 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14304180134241c792ec1f23ff89f94c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:43.166514 7154 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.167311 9151 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.170264 9419 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:43.170846 10276 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:43.171941 9419 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.172214 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 14304180134241c792ec1f23ff89f94c with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:43.172447 11572 tablet_replica.cc:330] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:43.172577 11572 raft_consensus.cc:2229] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.172811 11572 raft_consensus.cc:2258] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.173717 11572 ts_tablet_manager.cc:1875] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.176179 11572 ts_tablet_manager.cc:1888] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 13:59:43.176294 11572 log.cc:1196] T 14304180134241c792ec1f23ff89f94c P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/14304180134241c792ec1f23ff89f94c
I20240503 13:59:43.176824 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet 14304180134241c792ec1f23ff89f94c (table rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]) successfully deleted
I20240503 13:59:43.186492 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.191908 9151 raft_consensus.cc:1203] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:43.200223 10197 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.208457 7692 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.208995 7149 catalog_manager.cc:5484] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.210353 9417 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.214077 7299 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.239419 7427 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.244259 9283 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.244994 9417 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 13:59:43.245651 7367 consensus_peers.cc:485] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 902cf3db3581465c83170c4b13b3bffb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.245661 11197 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.245908 11602 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.248998 11197 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.251821 9283 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.252645 9417 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.254680 7152 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 9 to 10, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:43.268286 7297 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:43.270087 7562 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.270577 9024 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.270560 10386 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.272361 7692 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.272730 10386 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 13:59:43.273281 7234 consensus_peers.cc:485] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23cf5703ccef47ea927342b9c29f820a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.273463 10386 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.275663 7690 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:43.277276 7561 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.277565 9283 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.277832 10082 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.278026 10291 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 13:59:43.278662 7626 consensus_peers.cc:485] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): Couldn't send request to peer e1469ca6a5f945ec89ee1eb8df023b96. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.280267 10386 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.281152 9024 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.282074 10386 raft_consensus.cc:1050] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:43.282505 10386 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:43.286226 7690 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.286693 7692 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.287503 7152 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 4 to 5, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.287989 9024 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.288466 10752 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:59:43.289752 7234 consensus_peers.cc:485] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23cf5703ccef47ea927342b9c29f820a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.290127 10142 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.291960 7558 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.292606 10142 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.293360 7558 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.295537 10142 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.296072 10082 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.297049 9283 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.297498 7692 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.299949 7561 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.300256 7150 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.300335 7149 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 9 to 10, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.303681 7558 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.311517 9024 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.314175 7299 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:43.315609 9024 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.316012 10386 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.316073 7558 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:43.316215 7234 consensus_peers.cc:485] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23cf5703ccef47ea927342b9c29f820a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.316529 10386 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.319836 10386 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.321709 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23cf5703ccef47ea927342b9c29f820a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:43.321619 9024 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.322847 7150 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 6 to 7, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.323405 7558 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.338660 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cf5703ccef47ea927342b9c29f820a with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:43.338992 11572 tablet_replica.cc:330] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:43.339176 11572 raft_consensus.cc:2229] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.339426 11572 raft_consensus.cc:2258] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.339406 7299 consensus_queue.cc:237] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.340307 11572 ts_tablet_manager.cc:1875] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.341107 9151 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.341697 10752 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.342741 7427 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.343545 11572 ts_tablet_manager.cc:1888] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:43.343677 11572 log.cc:1196] T 23cf5703ccef47ea927342b9c29f820a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/23cf5703ccef47ea927342b9c29f820a
I20240503 13:59:43.344254 10752 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.344393 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet 23cf5703ccef47ea927342b9c29f820a (table rebalance_test_table_8 [id=3a09ea72b63549d08f82a8a8cd0841b8]) successfully deleted
I20240503 13:59:43.345000 7427 consensus_queue.cc:237] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.347065 7558 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.347396 9022 raft_consensus.cc:1261] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 13:59:43.347630 7367 consensus_peers.cc:485] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.347643 11160 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.348179 11160 consensus_queue.cc:1035] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 13:59:43.348671 7236 consensus_peers.cc:485] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f73a5db0cae647d4b3375e6d42358307. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.353717 9417 raft_consensus.cc:1203] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:43.357208 11716 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.357599 9151 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.360293 7154 catalog_manager.cc:5484] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.381206 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.386328 10299 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.388631 9024 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.389243 7558 raft_consensus.cc:2932] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.390966 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.392086 7155 catalog_manager.cc:5484] T 4e2cabe02dbf4580b53875aaa09aaffa P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 8 to 9, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.393291 7427 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.400825 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.416013 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.424230 7558 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
W20240503 13:59:43.426805 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade40ddaf0304b1a9e94db6630175c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.427215 9024 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.427927 7299 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.428606 10201 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.428750 11158 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.429241 9151 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.429764 10201 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.431578 11158 raft_consensus.cc:1013] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: attempt to promote peer 476f53c7e40c4a0b86ffbd818aea818d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 13:59:43.435125 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.441210 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.443692 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.464316 10201 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.467742 9024 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.471153 7151 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.471349 9151 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.471966 7299 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.473872 10197 raft_consensus.cc:1050] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:43.474390 10197 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.476773 9022 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.478165 7299 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.478567 11158 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.478865 11158 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 13:59:43.479944 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade40ddaf0304b1a9e94db6630175c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.482573 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.483079 9151 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.483538 10201 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.490875 11813 ts_tablet_manager.cc:902] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:43.491859 11813 tablet_copy_client.cc:308] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:43.492529 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 33b7b43742e34431af0c16d320b35dd7 from peer e1469ca6a5f945ec89ee1eb8df023b96 ({username='slave'} at 127.0.99.129:39677)
I20240503 13:59:43.492645 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 33b7b43742e34431af0c16d320b35dd7 from peer e1469ca6a5f945ec89ee1eb8df023b96 at {username='slave'} at 127.0.99.129:39677: session id = e1469ca6a5f945ec89ee1eb8df023b96-33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:43.492702 10197 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.493592 7450 tablet_copy_source_session.cc:205] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.494735 7299 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.494742 9022 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.496513 7151 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.498358 11813 ts_tablet_manager.cc:1875] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:43.501338 11813 ts_tablet_manager.cc:1888] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240503 13:59:43.501530 11813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b7b43742e34431af0c16d320b35dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.502887 11813 tablet_copy_client.cc:783] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.503096 11813 tablet_copy_client.cc:647] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.509614 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.510329 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.513475 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.513787 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.520298 9151 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.526415 7558 consensus_queue.cc:237] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.528172 7299 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 13:59:43.528519 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade40ddaf0304b1a9e94db6630175c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.528626 9022 raft_consensus.cc:1261] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.528623 11814 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.528714 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.530124 10197 consensus_queue.cc:1035] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.531646 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.531541 10197 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.532022 7299 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.534246 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ade40ddaf0304b1a9e94db6630175c0e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:43.534891 7149 catalog_manager.cc:5484] T ade40ddaf0304b1a9e94db6630175c0e P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 9 to 10, VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.536155 9022 raft_consensus.cc:2932] T ade40ddaf0304b1a9e94db6630175c0e P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.537778 9136 tablet_service.cc:1514] Processing DeleteTablet for tablet ade40ddaf0304b1a9e94db6630175c0e with delete_type TABLET_DATA_TOMBSTONED (TS 708734e6729441fcb2c5952fa47771d9 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56318
I20240503 13:59:43.537909 11813 tablet_copy_client.cc:523] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.539983 11813 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:43.555534 7690 consensus_queue.cc:237] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.556965 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.557847 7299 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.558531 9283 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.558786 11717 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.559876 7558 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.560370 11717 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 13:59:43.562325 7629 consensus_peers.cc:485] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93d454886dfa40018a4403a3d74b3de2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.562530 11717 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.564499 11717 raft_consensus.cc:1013] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: attempt to promote peer c5c3042421284f028fa512b7876da405: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 13:59:43.565876 7427 consensus_queue.cc:237] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
W20240503 13:59:43.567884 7364 consensus_peers.cc:485] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 -> Peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): Couldn't send request to peer e1469ca6a5f945ec89ee1eb8df023b96. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.568526 9283 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 13:59:43.568531 9417 raft_consensus.cc:1261] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 13:59:43.569015 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:43.569312 11160 consensus_queue.cc:1035] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:43.571323 11197 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.573819 9283 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.573926 9417 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.575616 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33b7b43742e34431af0c16d320b35dd7 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:43.577009 7158 catalog_manager.cc:5484] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 13 to 14, VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:43.578634 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.596263 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.600455 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.608566 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.614187 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.614275 11829 tablet_replica.cc:330] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: stopping tablet replica
I20240503 13:59:43.614436 11829 raft_consensus.cc:2229] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.614612 11829 raft_consensus.cc:2258] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.615399 11829 ts_tablet_manager.cc:1875] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.620638 11829 ts_tablet_manager.cc:1888] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 13:59:43.622360 9005 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b7b43742e34431af0c16d320b35dd7 with delete_type TABLET_DATA_TOMBSTONED (TS 476f53c7e40c4a0b86ffbd818aea818d not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:44488
I20240503 13:59:43.623658 11829 log.cc:1196] T ade40ddaf0304b1a9e94db6630175c0e P 708734e6729441fcb2c5952fa47771d9: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ade40ddaf0304b1a9e94db6630175c0e
I20240503 13:59:43.636473 11853 tablet_replica.cc:330] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: stopping tablet replica
I20240503 13:59:43.643915 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:43.638154 11853 raft_consensus.cc:2229] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.645370 11853 raft_consensus.cc:2258] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.645567 10291 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.648317 9283 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.649397 7156 catalog_manager.cc:5484] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.649493 7299 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.652956 11717 raft_consensus.cc:1050] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:43.653401 11717 consensus_queue.cc:237] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.654029 11853 ts_tablet_manager.cc:1875] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.656338 9283 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 13:59:43.656404 7629 consensus_peers.cc:485] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93d454886dfa40018a4403a3d74b3de2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.656364 7558 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.657021 7299 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.658885 11608 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:43.659775 7145 catalog_manager.cc:4830] TS 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): tablet ade40ddaf0304b1a9e94db6630175c0e (table rebalance_test_table_5 [id=6bab62a3656442a6a25a416f35f12e63]) successfully deleted
I20240503 13:59:43.661518 7561 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.662731 11608 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:43.663018 11608 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:43.666952 11860 ts_tablet_manager.cc:902] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:43.667155 10291 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.667564 11860 tablet_copy_client.cc:308] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:43.668113 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 902cf3db3581465c83170c4b13b3bffb from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:47471)
I20240503 13:59:43.668226 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 902cf3db3581465c83170c4b13b3bffb from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:47471: session id = 708734e6729441fcb2c5952fa47771d9-902cf3db3581465c83170c4b13b3bffb
I20240503 13:59:43.668228 11853 ts_tablet_manager.cc:1888] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240503 13:59:43.668325 11853 log.cc:1196] T 33b7b43742e34431af0c16d320b35dd7 P 476f53c7e40c4a0b86ffbd818aea818d: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/33b7b43742e34431af0c16d320b35dd7
I20240503 13:59:43.669178 7450 tablet_copy_source_session.cc:205] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.669826 9283 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.670908 7561 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.671461 7149 catalog_manager.cc:5484] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.673085 11860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902cf3db3581465c83170c4b13b3bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.674141 11861 ts_tablet_manager.cc:902] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:43.674381 7299 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.674824 11861 tablet_copy_client.cc:308] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:43.675060 7690 consensus_queue.cc:237] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.675330 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet ff246d9b2c594282bf2e99d2fd2b617d from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:58515)
I20240503 13:59:43.675417 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet ff246d9b2c594282bf2e99d2fd2b617d from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:58515: session id = 708734e6729441fcb2c5952fa47771d9-ff246d9b2c594282bf2e99d2fd2b617d
I20240503 13:59:43.676450 7710 tablet_copy_source_session.cc:205] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.678253 9283 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:43.678283 7629 consensus_peers.cc:485] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93d454886dfa40018a4403a3d74b3de2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.678737 11717 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.678531 7299 raft_consensus.cc:1261] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.679607 11717 consensus_queue.cc:1035] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.680356 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.681464 11717 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.682147 11860 tablet_copy_client.cc:783] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.682380 11860 tablet_copy_client.cc:647] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.682425 7297 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.684163 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:43.686880 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93d454886dfa40018a4403a3d74b3de2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:43.688191 9283 raft_consensus.cc:2932] T 93d454886dfa40018a4403a3d74b3de2 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:43.689095 7155 catalog_manager.cc:5484] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.692665 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d454886dfa40018a4403a3d74b3de2 with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37520
I20240503 13:59:43.693173 11861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff246d9b2c594282bf2e99d2fd2b617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.695873 7145 catalog_manager.cc:4830] TS 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): tablet 33b7b43742e34431af0c16d320b35dd7 (table rebalance_test_table_4 [id=b8b322e08e5d48f39595d2b2ee79d218]) successfully deleted
I20240503 13:59:43.701562 11861 tablet_copy_client.cc:783] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.701866 11861 tablet_copy_client.cc:647] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.709688 11860 tablet_copy_client.cc:523] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.710373 11861 tablet_copy_client.cc:523] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.711310 11877 ts_tablet_manager.cc:902] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:43.711849 11877 tablet_copy_client.cc:308] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:43.712400 11861 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:43.714522 11860 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:43.715517 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 14304180134241c792ec1f23ff89f94c from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:58407)
I20240503 13:59:43.715623 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 14304180134241c792ec1f23ff89f94c from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:58407: session id = 476f53c7e40c4a0b86ffbd818aea818d-14304180134241c792ec1f23ff89f94c
I20240503 13:59:43.716516 7320 tablet_copy_source_session.cc:205] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.717496 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14304180134241c792ec1f23ff89f94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.721307 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.722611 11875 tablet_replica.cc:330] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: stopping tablet replica
I20240503 13:59:43.725064 11875 raft_consensus.cc:2229] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.725415 11875 raft_consensus.cc:2258] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.731071 11875 ts_tablet_manager.cc:1875] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.733618 11875 ts_tablet_manager.cc:1888] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:43.733947 11875 log.cc:1196] T 93d454886dfa40018a4403a3d74b3de2 P 9d921974718b4613b136eee9d0e1bad2: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/93d454886dfa40018a4403a3d74b3de2
I20240503 13:59:43.737232 11876 ts_tablet_manager.cc:902] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:43.737954 11876 tablet_copy_client.cc:308] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:43.739302 11877 tablet_copy_client.cc:783] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.739598 11877 tablet_copy_client.cc:647] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.743528 11813 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.743877 11717 raft_consensus.cc:1050] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:43.738657 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet a1115afa5ead4c95aea379ad9987cbe0 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:40203)
I20240503 13:59:43.744256 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet a1115afa5ead4c95aea379ad9987cbe0 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:40203: session id = c5c3042421284f028fa512b7876da405-a1115afa5ead4c95aea379ad9987cbe0
I20240503 13:59:43.744372 11717 consensus_queue.cc:237] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:43.745281 7582 tablet_copy_source_session.cc:205] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.745999 7427 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.746239 9151 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.746591 10291 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.747411 10291 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.747665 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1115afa5ead4c95aea379ad9987cbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.750960 7142 catalog_manager.cc:4830] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): tablet 93d454886dfa40018a4403a3d74b3de2 (table rebalance_test_table_7 [id=a5274a33c27e4111801d2bb4649f3d1a]) successfully deleted
I20240503 13:59:43.751696 11876 tablet_copy_client.cc:783] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.751895 11876 tablet_copy_client.cc:647] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.746891 7299 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:43.756937 11813 tablet_bootstrap.cc:492] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap complete.
I20240503 13:59:43.756937 10291 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:43.757172 11813 ts_tablet_manager.cc:1367] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.217s user 0.030s sys 0.000s
I20240503 13:59:43.757686 11813 raft_consensus.cc:348] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:43.757822 11813 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: LEARNER
I20240503 13:59:43.757920 11813 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:43.758675 7297 consensus_queue.cc:237] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.761448 11877 tablet_copy_client.cc:523] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.763183 11877 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:43.764426 11876 tablet_copy_client.cc:523] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.766103 11876 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:43.768018 11861 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.768421 11861 tablet_bootstrap.cc:492] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:43.768637 11861 ts_tablet_manager.cc:1367] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.056s user 0.029s sys 0.000s
I20240503 13:59:43.769156 11861 raft_consensus.cc:348] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.769366 11861 raft_consensus.cc:729] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:43.769619 11861 consensus_queue.cc:260] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.771062 11813 ts_tablet_manager.cc:1398] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 13:59:43.771680 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session e1469ca6a5f945ec89ee1eb8df023b96-33b7b43742e34431af0c16d320b35dd7 received from {username='slave'} at 127.0.99.129:39677
I20240503 13:59:43.771785 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session e1469ca6a5f945ec89ee1eb8df023b96-33b7b43742e34431af0c16d320b35dd7 on tablet 33b7b43742e34431af0c16d320b35dd7 with peer e1469ca6a5f945ec89ee1eb8df023b96
I20240503 13:59:43.775077 11855 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.778180 11911 ts_tablet_manager.cc:902] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:43.778393 7299 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.778779 11911 tablet_copy_client.cc:308] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:43.779240 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet f73a5db0cae647d4b3375e6d42358307 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:58407)
I20240503 13:59:43.779333 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet f73a5db0cae647d4b3375e6d42358307 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:58407: session id = 476f53c7e40c4a0b86ffbd818aea818d-f73a5db0cae647d4b3375e6d42358307
I20240503 13:59:43.780287 7320 tablet_copy_source_session.cc:205] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.780375 11897 ts_tablet_manager.cc:902] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:43.780966 11897 tablet_copy_client.cc:308] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:43.781098 7427 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.781184 9151 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.782974 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73a5db0cae647d4b3375e6d42358307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.783109 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet 4e2cabe02dbf4580b53875aaa09aaffa from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:53079)
I20240503 13:59:43.783201 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet 4e2cabe02dbf4580b53875aaa09aaffa from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:53079: session id = c5c3042421284f028fa512b7876da405-4e2cabe02dbf4580b53875aaa09aaffa
I20240503 13:59:43.784075 7450 tablet_copy_source_session.cc:205] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.785547 7155 catalog_manager.cc:5484] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 4 to 5, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:43.789095 11901 ts_tablet_manager.cc:902] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:43.789920 11901 tablet_copy_client.cc:308] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:43.790338 7427 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.790599 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 23cf5703ccef47ea927342b9c29f820a from peer 1e5c0b4b872d451dba60797739f83576 ({username='slave'} at 127.0.99.130:50713)
I20240503 13:59:43.790715 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 23cf5703ccef47ea927342b9c29f820a from peer 1e5c0b4b872d451dba60797739f83576 at {username='slave'} at 127.0.99.130:50713: session id = 1e5c0b4b872d451dba60797739f83576-23cf5703ccef47ea927342b9c29f820a
I20240503 13:59:43.790735 9417 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.790755 10385 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.791494 10385 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.791648 7320 tablet_copy_source_session.cc:205] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.793045 10752 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.793561 11901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23cf5703ccef47ea927342b9c29f820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.793871 11861 ts_tablet_manager.cc:1398] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20240503 13:59:43.796200 11901 tablet_copy_client.cc:783] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.796391 11901 tablet_copy_client.cc:647] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.796578 9419 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.796787 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-ff246d9b2c594282bf2e99d2fd2b617d received from {username='slave'} at 127.0.99.134:58515
I20240503 13:59:43.796867 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-ff246d9b2c594282bf2e99d2fd2b617d on tablet ff246d9b2c594282bf2e99d2fd2b617d with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:43.798913 7427 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.799335 7158 catalog_manager.cc:5484] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.803941 11911 tablet_copy_client.cc:783] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.804199 11911 tablet_copy_client.cc:647] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.807109 11911 tablet_copy_client.cc:523] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.808786 11911 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
W20240503 13:59:43.814257 7236 consensus_peers.cc:485] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b0fd771f844fcab38aec069b9abd00. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.821764 11901 tablet_copy_client.cc:523] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.822319 11897 ts_tablet_manager.cc:1875] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:43.823961 11901 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
I20240503 13:59:43.824054 11860 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.824251 7690 consensus_queue.cc:237] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:43.824563 11860 tablet_bootstrap.cc:492] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:43.824808 11860 ts_tablet_manager.cc:1367] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.110s user 0.027s sys 0.003s
I20240503 13:59:43.825322 11860 raft_consensus.cc:348] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.825515 11860 raft_consensus.cc:729] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:43.825663 11860 consensus_queue.cc:260] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:43.826754 11860 ts_tablet_manager.cc:1398] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:43.827232 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-902cf3db3581465c83170c4b13b3bffb received from {username='slave'} at 127.0.99.134:47471
I20240503 13:59:43.827307 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-902cf3db3581465c83170c4b13b3bffb on tablet 902cf3db3581465c83170c4b13b3bffb with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:43.828851 11897 ts_tablet_manager.cc:1888] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 13:59:43.829073 11897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2cabe02dbf4580b53875aaa09aaffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.830562 7297 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.831499 10291 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:43.832749 7427 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.833356 9151 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:43.833963 11608 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:43.834421 11608 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:59:43.834955 7629 consensus_peers.cc:485] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b04204ad281452794d724fb4d7212f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.837445 11897 tablet_copy_client.cc:783] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.837704 11897 tablet_copy_client.cc:647] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.837672 10291 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.839038 11897 tablet_copy_client.cc:523] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.840524 11897 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:43.841607 7156 catalog_manager.cc:5484] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.844380 9151 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.844416 7427 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.850085 11819 raft_consensus.cc:1050] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER da390a1a407a4f30b217b3d60fba563f to VOTER
I20240503 13:59:43.850548 11819 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:43.846513 11877 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.852864 11877 tablet_bootstrap.cc:492] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:43.853077 9022 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.853169 11877 ts_tablet_manager.cc:1367] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.090s user 0.029s sys 0.000s
I20240503 13:59:43.853619 11877 raft_consensus.cc:348] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.853824 11877 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:43.853888 10201 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.854012 11877 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.854669 9419 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.854859 11877 ts_tablet_manager.cc:1398] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:43.855362 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-14304180134241c792ec1f23ff89f94c received from {username='slave'} at 127.0.99.133:58407
I20240503 13:59:43.855445 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-14304180134241c792ec1f23ff89f94c on tablet 14304180134241c792ec1f23ff89f94c with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:43.857417 10201 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.858502 7299 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.859087 11911 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.859329 11911 tablet_bootstrap.cc:492] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:43.859498 11911 ts_tablet_manager.cc:1367] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.004s
I20240503 13:59:43.859961 11911 raft_consensus.cc:348] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.860064 11911 raft_consensus.cc:729] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:43.860158 11911 consensus_queue.cc:260] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:43.860284 11813 ts_tablet_manager.cc:902] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:43.860528 11911 ts_tablet_manager.cc:1398] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:43.860733 11813 tablet_copy_client.cc:308] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:43.861004 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-f73a5db0cae647d4b3375e6d42358307 received from {username='slave'} at 127.0.99.133:58407
I20240503 13:59:43.861068 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-f73a5db0cae647d4b3375e6d42358307 on tablet f73a5db0cae647d4b3375e6d42358307 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:43.861929 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet b23f9eeeb4af45b7b679b70514e247b8 from peer e1469ca6a5f945ec89ee1eb8df023b96 ({username='slave'} at 127.0.99.129:43167)
I20240503 13:59:43.862746 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet b23f9eeeb4af45b7b679b70514e247b8 from peer e1469ca6a5f945ec89ee1eb8df023b96 at {username='slave'} at 127.0.99.129:43167: session id = e1469ca6a5f945ec89ee1eb8df023b96-b23f9eeeb4af45b7b679b70514e247b8
I20240503 13:59:43.863777 7710 tablet_copy_source_session.cc:205] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.865160 11813 ts_tablet_manager.cc:1875] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:43.866487 7297 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.866919 10201 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.867630 11813 ts_tablet_manager.cc:1888] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 13:59:43.867815 11813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23f9eeeb4af45b7b679b70514e247b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.868360 7690 consensus_queue.cc:237] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:43.868952 11813 tablet_copy_client.cc:783] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.869086 11813 tablet_copy_client.cc:647] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.870388 7427 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.870893 11855 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.871730 11876 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.872038 11876 tablet_bootstrap.cc:492] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:43.872200 11876 ts_tablet_manager.cc:1367] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.106s user 0.023s sys 0.003s
I20240503 13:59:43.872648 11876 raft_consensus.cc:348] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.872778 11876 raft_consensus.cc:729] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:43.872887 11876 consensus_queue.cc:260] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.873262 11876 ts_tablet_manager.cc:1398] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.874197 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session c5c3042421284f028fa512b7876da405-a1115afa5ead4c95aea379ad9987cbe0 received from {username='slave'} at 127.0.99.135:40203
I20240503 13:59:43.874300 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session c5c3042421284f028fa512b7876da405-a1115afa5ead4c95aea379ad9987cbe0 on tablet a1115afa5ead4c95aea379ad9987cbe0 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:43.874282 11717 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.874773 9152 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.874717 7427 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.878064 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b04204ad281452794d724fb4d7212f8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:43.878273 11855 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.878381 7151 catalog_manager.cc:5484] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:43.877964 7629 consensus_peers.cc:485] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b04204ad281452794d724fb4d7212f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.881152 9152 raft_consensus.cc:2932] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.881896 11814 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:43.882908 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b04204ad281452794d724fb4d7212f8 with delete_type TABLET_DATA_TOMBSTONED (TS e1469ca6a5f945ec89ee1eb8df023b96 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45016
I20240503 13:59:43.884861 9419 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:43.884891 9022 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:43.888223 7156 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 7 to 8, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:43.889190 7297 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da390a1a407a4f30b217b3d60fba563f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:43.893491 7561 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.895144 9152 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:43.895246 7690 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 13:59:43.895303 7498 consensus_peers.cc:485] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa9fb136138439b8f694ac95dbfa2e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.895668 10201 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.895869 11819 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:43.899840 7297 consensus_queue.cc:237] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.901878 11819 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.904174 11964 tablet_replica.cc:330] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: stopping tablet replica
I20240503 13:59:43.904616 7690 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.905036 7155 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 9 to 10, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.905380 11964 raft_consensus.cc:2229] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.905596 11964 raft_consensus.cc:2258] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 13:59:43.908612 7236 consensus_peers.cc:485] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3622884b9b314ccca24422d8fa05360a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.908628 9419 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.908778 7561 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.909049 10142 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.909224 11716 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.910282 11876 ts_tablet_manager.cc:902] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:43.910781 11876 tablet_copy_client.cc:308] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:43.911267 11964 ts_tablet_manager.cc:1875] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.912040 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet ade40ddaf0304b1a9e94db6630175c0e from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:40203)
I20240503 13:59:43.912137 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet ade40ddaf0304b1a9e94db6630175c0e from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:40203: session id = c5c3042421284f028fa512b7876da405-ade40ddaf0304b1a9e94db6630175c0e
I20240503 13:59:43.913034 7582 tablet_copy_source_session.cc:205] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:43.914494 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade40ddaf0304b1a9e94db6630175c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:43.918738 9152 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.919862 7690 consensus_queue.cc:237] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:43.920097 11876 tablet_copy_client.cc:783] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:43.920253 11876 tablet_copy_client.cc:647] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:43.920693 11813 tablet_copy_client.cc:523] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.921833 7427 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.922271 11855 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.922421 11813 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:43.926614 9283 raft_consensus.cc:1261] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:43.926781 7629 consensus_peers.cc:485] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4582c127b7d34e08a68ea9094de11f19. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.927007 11855 consensus_queue.cc:1035] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.929199 11855 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.929993 11876 tablet_copy_client.cc:523] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:43.930284 10142 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.931485 11876 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:43.932096 7427 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.933284 9419 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.935640 7156 catalog_manager.cc:5484] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.936481 7561 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:43.936753 7155 catalog_manager.cc:5484] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.936960 9283 raft_consensus.cc:2932] T 4582c127b7d34e08a68ea9094de11f19 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:43.942392 11964 ts_tablet_manager.cc:1888] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 13:59:43.942492 11964 log.cc:1196] T 9b04204ad281452794d724fb4d7212f8 P e1469ca6a5f945ec89ee1eb8df023b96: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b04204ad281452794d724fb4d7212f8
I20240503 13:59:43.943554 7142 catalog_manager.cc:4830] TS e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): tablet 9b04204ad281452794d724fb4d7212f8 (table rebalance_test_table_6 [id=581ba72cb36f49fabf7a67fb8d8d47ef]) successfully deleted
I20240503 13:59:43.950976 7690 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } }
W20240503 13:59:43.955245 7626 consensus_peers.cc:485] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): Couldn't send request to peer 9d921974718b4613b136eee9d0e1bad2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff246d9b2c594282bf2e99d2fd2b617d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.955821 9152 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.955868 9419 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.956346 10291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.956667 10291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.959048 7427 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:43.959492 11608 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.963209 7156 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 7 to 8, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:43.964428 10291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:43.965276 9419 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.966271 7561 consensus_queue.cc:237] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:43.968755 7690 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:43.969395 7431 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.970433 9151 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:43.970615 7496 consensus_peers.cc:485] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.970968 10201 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.971416 10201 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:43.975242 11813 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.975525 11813 tablet_bootstrap.cc:492] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap complete.
I20240503 13:59:43.975711 11813 ts_tablet_manager.cc:1367] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.053s user 0.022s sys 0.003s
I20240503 13:59:43.976229 11813 raft_consensus.cc:348] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:43.976357 11813 raft_consensus.cc:729] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: LEARNER
I20240503 13:59:43.976444 11813 consensus_queue.cc:260] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:43.976492 10201 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.976804 11813 ts_tablet_manager.cc:1398] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:43.978410 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session e1469ca6a5f945ec89ee1eb8df023b96-b23f9eeeb4af45b7b679b70514e247b8 received from {username='slave'} at 127.0.99.129:43167
I20240503 13:59:43.978488 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session e1469ca6a5f945ec89ee1eb8df023b96-b23f9eeeb4af45b7b679b70514e247b8 on tablet b23f9eeeb4af45b7b679b70514e247b8 with peer e1469ca6a5f945ec89ee1eb8df023b96
I20240503 13:59:43.979305 11901 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:43.979396 7690 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.979650 11901 tablet_bootstrap.cc:492] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: Bootstrap complete.
I20240503 13:59:43.982611 11717 raft_consensus.cc:1050] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:43.982952 9152 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.983073 11717 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } }
I20240503 13:59:43.983130 7155 catalog_manager.cc:5484] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:43.984892 7431 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.985853 11855 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.986407 9419 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.986361 9152 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.988018 9151 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:43.990289 11855 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.990593 11855 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:43.990693 11901 ts_tablet_manager.cc:1367] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.167s user 0.026s sys 0.005s
W20240503 13:59:43.991149 7626 consensus_peers.cc:485] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): Couldn't send request to peer 9d921974718b4613b136eee9d0e1bad2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff246d9b2c594282bf2e99d2fd2b617d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:43.991402 11901 raft_consensus.cc:348] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:43.991529 11901 raft_consensus.cc:729] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: LEARNER
I20240503 13:59:43.991641 11901 consensus_queue.cc:260] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:43.991923 11901 ts_tablet_manager.cc:1398] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:43.992458 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 1e5c0b4b872d451dba60797739f83576-23cf5703ccef47ea927342b9c29f820a received from {username='slave'} at 127.0.99.130:50713
I20240503 13:59:43.992537 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 1e5c0b4b872d451dba60797739f83576-23cf5703ccef47ea927342b9c29f820a on tablet 23cf5703ccef47ea927342b9c29f820a with peer 1e5c0b4b872d451dba60797739f83576
I20240503 13:59:43.993942 10291 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.994577 9419 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.997167 7431 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.998121 7156 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 8 to 9, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:43.998317 9151 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:44.005247 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:44.006613 7690 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:44.009251 9151 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.009656 9419 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.009918 7561 consensus_queue.cc:237] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:44.010231 10082 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.010519 10082 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.011965 9285 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:44.012022 7690 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:44.012269 7297 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:44.012431 11814 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:44.012696 11819 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:44.013518 11819 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:44.017485 11608 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:44.019250 11158 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
W20240503 13:59:44.019526 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.020386 9419 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:44.021328 7143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff246d9b2c594282bf2e99d2fd2b617d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:44.022462 7300 raft_consensus.cc:2932] T 33b7b43742e34431af0c16d320b35dd7 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:44.024094 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.024116 9151 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:44.026865 7690 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:44.026932 7156 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 9 to 10, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:44.029803 7158 catalog_manager.cc:5484] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.030499 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet ff246d9b2c594282bf2e99d2fd2b617d with delete_type TABLET_DATA_TOMBSTONED (TS 9d921974718b4613b136eee9d0e1bad2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37520
W20240503 13:59:44.031026 7142 catalog_manager.cc:4797] TS 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): delete failed for tablet ff246d9b2c594282bf2e99d2fd2b617d because the tablet was not found. No further retry: Not found: Tablet not found: ff246d9b2c594282bf2e99d2fd2b617d
I20240503 13:59:44.032598 7297 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:44.037814 7431 raft_consensus.cc:1261] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.037842 7561 consensus_queue.cc:237] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.038466 10291 consensus_queue.cc:1035] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 13:59:44.041301 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:44.041465 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.041750 7690 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:44.042052 7431 raft_consensus.cc:2932] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:44.042199 11819 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:44.046265 7297 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:44.047674 11814 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.049536 10197 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:44.050765 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.050665 7690 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.050957 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:44.054211 7150 catalog_manager.cc:5484] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.058820 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:44.059312 7297 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.072614 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.078630 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.105525 9151 raft_consensus.cc:1203] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 13:59:44.110177 7561 consensus_queue.cc:237] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.112110 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:44.113742 9022 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.113762 9283 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:44.114162 7297 raft_consensus.cc:1261] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:44.114624 7498 consensus_peers.cc:485] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86634d81241c44cebcb602a6982f4244. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.114869 11819 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.114917 11158 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:44.115211 11819 consensus_queue.cc:1035] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.118875 11897 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.119298 11897 tablet_bootstrap.cc:492] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:44.121012 9283 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:44.128134 9283 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.130228 11897 ts_tablet_manager.cc:1367] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.290s user 0.028s sys 0.003s
I20240503 13:59:44.130796 11897 raft_consensus.cc:348] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:44.131067 11897 raft_consensus.cc:729] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:44.131237 11897 consensus_queue.cc:260] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:44.131597 11158 raft_consensus.cc:1050] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER c5c3042421284f028fa512b7876da405 to VOTER
I20240503 13:59:44.131738 11897 ts_tablet_manager.cc:1398] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.132117 11158 consensus_queue.cc:237] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.132200 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session c5c3042421284f028fa512b7876da405-4e2cabe02dbf4580b53875aaa09aaffa received from {username='slave'} at 127.0.99.135:53079
I20240503 13:59:44.132287 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session c5c3042421284f028fa512b7876da405-4e2cabe02dbf4580b53875aaa09aaffa on tablet 4e2cabe02dbf4580b53875aaa09aaffa with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:44.133698 9283 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:44.134251 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:44.134440 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.134450 7690 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.134523 11819 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.134990 11819 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.135851 7297 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.135876 7561 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:44.137914 9024 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 13:59:44.138213 7496 consensus_peers.cc:485] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.138486 11814 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.138759 9419 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:44.139043 7690 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.140076 11814 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:44.140445 11814 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:44.142097 7297 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:44.142093 10201 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.142580 10201 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:44.143548 7297 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.143746 9283 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.145874 11158 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.146251 9419 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.147555 7155 catalog_manager.cc:5484] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 6 to 7, c5c3042421284f028fa512b7876da405 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.149545 9024 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.149674 7431 consensus_queue.cc:237] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:44.150177 7297 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
W20240503 13:59:44.151374 7367 consensus_peers.cc:485] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.152938 9283 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:44.153609 11602 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:44.155019 7690 raft_consensus.cc:1261] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:44.156737 11602 consensus_queue.cc:1035] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:44.157248 11876 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.157526 11876 tablet_bootstrap.cc:492] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:44.157697 11876 ts_tablet_manager.cc:1367] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.226s user 0.026s sys 0.000s
I20240503 13:59:44.159651 7150 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 8 to 9, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.166471 11876 raft_consensus.cc:348] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:44.166639 11876 raft_consensus.cc:729] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:44.166752 11876 consensus_queue.cc:260] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:44.170850 9024 raft_consensus.cc:1203] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 13:59:44.174746 11876 ts_tablet_manager.cc:1398] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240503 13:59:44.175745 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session c5c3042421284f028fa512b7876da405-ade40ddaf0304b1a9e94db6630175c0e received from {username='slave'} at 127.0.99.135:40203
I20240503 13:59:44.175843 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session c5c3042421284f028fa512b7876da405-ade40ddaf0304b1a9e94db6630175c0e on tablet ade40ddaf0304b1a9e94db6630175c0e with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:44.176827 7561 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:44.178310 9417 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.178764 9024 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.179206 11814 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.179528 11814 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.180006 7297 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.180478 10197 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.182694 11814 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.183545 9024 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.184437 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 110ac05b2e8c4594a03225ab2c1ad56d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:44.185819 11911 ts_tablet_manager.cc:902] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:44.185959 9417 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.186327 11911 tablet_copy_client.cc:308] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:44.186826 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 93d454886dfa40018a4403a3d74b3de2 from peer 476f53c7e40c4a0b86ffbd818aea818d ({username='slave'} at 127.0.99.133:54357)
I20240503 13:59:44.186960 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 93d454886dfa40018a4403a3d74b3de2 from peer 476f53c7e40c4a0b86ffbd818aea818d at {username='slave'} at 127.0.99.133:54357: session id = 476f53c7e40c4a0b86ffbd818aea818d-93d454886dfa40018a4403a3d74b3de2
I20240503 13:59:44.188020 7710 tablet_copy_source_session.cc:205] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:44.189080 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d454886dfa40018a4403a3d74b3de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.188974 7297 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.191804 11911 tablet_copy_client.cc:783] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:44.191969 11911 tablet_copy_client.cc:647] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:44.193722 7150 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:44.197741 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 110ac05b2e8c4594a03225ab2c1ad56d with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:44.197537 11911 tablet_copy_client.cc:523] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:44.198359 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet 110ac05b2e8c4594a03225ab2c1ad56d (table rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]) successfully deleted
I20240503 13:59:44.200289 11911 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap starting.
I20240503 13:59:44.206799 7561 consensus_queue.cc:237] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:44.208704 7431 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.208814 9285 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.209266 10197 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.209455 10201 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 13:59:44.209678 7498 consensus_peers.cc:485] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 708734e6729441fcb2c5952fa47771d9 (127.0.99.134:36383): Couldn't send request to peer 708734e6729441fcb2c5952fa47771d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491ee8399ef04ae7b93e3228fef05eb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.211474 10201 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.211892 7431 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.214774 7155 catalog_manager.cc:5484] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.216435 10197 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.218984 9022 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.221994 7157 catalog_manager.cc:5484] T 86634d81241c44cebcb602a6982f4244 P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.222085 11197 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.224179 7300 raft_consensus.cc:2932] T 86634d81241c44cebcb602a6982f4244 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.225106 9283 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.227958 7150 catalog_manager.cc:5484] T fca185e6c04648c5aa4f4ad51b0e1a39 P c5c3042421284f028fa512b7876da405 reported cstate change: config changed from index 8 to 9, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.233428 7690 raft_consensus.cc:2932] T fca185e6c04648c5aa4f4ad51b0e1a39 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.237572 9283 raft_consensus.cc:1203] T 4e2cabe02dbf4580b53875aaa09aaffa P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 13:59:44.238533 7300 raft_consensus.cc:1203] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 13:59:44.246299 11911 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.246847 11911 tablet_bootstrap.cc:492] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: Bootstrap complete.
I20240503 13:59:44.247753 11911 ts_tablet_manager.cc:1367] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent bootstrapping tablet: real 0.048s user 0.027s sys 0.000s
I20240503 13:59:44.248513 11911 raft_consensus.cc:348] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:44.248664 11911 raft_consensus.cc:729] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 476f53c7e40c4a0b86ffbd818aea818d, State: Initialized, Role: LEARNER
I20240503 13:59:44.248773 11911 consensus_queue.cc:260] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:44.248910 7561 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:44.249053 11911 ts_tablet_manager.cc:1398] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:44.250633 7300 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:44.251101 12069 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:44.251197 9022 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:44.251547 9417 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:44.251785 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-93d454886dfa40018a4403a3d74b3de2 received from {username='slave'} at 127.0.99.133:54357
I20240503 13:59:44.251860 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session 476f53c7e40c4a0b86ffbd818aea818d-93d454886dfa40018a4403a3d74b3de2 on tablet 93d454886dfa40018a4403a3d74b3de2 with peer 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:44.253366 12069 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:44.255311 12069 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 13:59:44.255668 7496 consensus_peers.cc:485] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.255861 10201 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.257160 9022 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.257654 7690 consensus_queue.cc:237] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:44.258692 7297 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.259402 9022 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.259743 9283 raft_consensus.cc:1261] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 13:59:44.259833 11717 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.260535 9417 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.261536 7158 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 10 to 11, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
W20240503 13:59:44.262881 7627 consensus_peers.cc:485] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.263749 10291 consensus_queue.cc:1035] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:44.264307 10291 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.266814 7431 consensus_queue.cc:237] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.268481 9151 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.269470 7690 raft_consensus.cc:1261] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.269767 9283 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } } }
I20240503 13:59:44.270041 7561 consensus_queue.cc:237] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:44.270177 9022 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.271603 7297 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.271965 9022 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.272089 10197 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.272397 10197 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.272527 11602 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.272792 7153 catalog_manager.cc:5484] T e0b7095bdf3549fb83b3573a8e65deee P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.274156 11602 consensus_queue.cc:1035] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.275095 12069 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.275570 9022 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.277348 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 110ac05b2e8c4594a03225ab2c1ad56d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:44.277451 9417 raft_consensus.cc:1261] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.277791 9283 raft_consensus.cc:2932] T e0b7095bdf3549fb83b3573a8e65deee P c5c3042421284f028fa512b7876da405 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } } }
I20240503 13:59:44.277822 7297 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
W20240503 13:59:44.280604 7367 consensus_peers.cc:485] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dffd8718851475099a6696e4a2b8901. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.280836 10197 consensus_queue.cc:1035] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.280969 7150 catalog_manager.cc:5484] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 11 to 12, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.283989 9417 raft_consensus.cc:2932] T 110ac05b2e8c4594a03225ab2c1ad56d P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } }
I20240503 13:59:44.284711 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 110ac05b2e8c4594a03225ab2c1ad56d with delete_type TABLET_DATA_TOMBSTONED (TS 1e5c0b4b872d451dba60797739f83576 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:39032
I20240503 13:59:44.285142 7143 catalog_manager.cc:4830] TS 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): tablet 110ac05b2e8c4594a03225ab2c1ad56d (table rebalance_test_table_9 [id=371f6713b69b4d2baaff6174ec18e696]) successfully deleted
I20240503 13:59:44.287626 7431 consensus_queue.cc:237] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:44.289606 9152 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.290148 11602 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.290685 12095 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
W20240503 13:59:44.291646 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.292286 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.293699 9417 raft_consensus.cc:1261] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.294180 11602 consensus_queue.cc:1035] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.297917 9136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.298215 11197 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:44.303804 9151 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.304451 7690 raft_consensus.cc:2932] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.307194 7158 catalog_manager.cc:5484] T 4dffd8718851475099a6696e4a2b8901 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 6 to 7, NON_VOTER da390a1a407a4f30b217b3d60fba563f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.307490 7155 catalog_manager.cc:5484] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 11 to 12, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.309218 9152 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:44.313158 9417 raft_consensus.cc:2932] T 2212cf72b6ee4ee5b086455a75482e3b P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c5c3042421284f028fa512b7876da405 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } }
I20240503 13:59:44.313447 7672 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:44.317735 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:44.327111 9267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.335220 7431 raft_consensus.cc:1203] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:44.340358 11876 ts_tablet_manager.cc:902] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:44.343880 11876 tablet_copy_client.cc:308] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:44.344383 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet faa9fb136138439b8f694ac95dbfa2e7 from peer c5c3042421284f028fa512b7876da405 ({username='slave'} at 127.0.99.135:40203)
I20240503 13:59:44.344491 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet faa9fb136138439b8f694ac95dbfa2e7 from peer c5c3042421284f028fa512b7876da405 at {username='slave'} at 127.0.99.135:40203: session id = c5c3042421284f028fa512b7876da405-faa9fb136138439b8f694ac95dbfa2e7
I20240503 13:59:44.345436 7582 tablet_copy_source_session.cc:205] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:44.346421 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa9fb136138439b8f694ac95dbfa2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.348626 11876 tablet_copy_client.cc:783] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:44.348803 11876 tablet_copy_client.cc:647] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:44.349889 11876 tablet_copy_client.cc:523] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:44.351354 11876 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: Bootstrap starting.
I20240503 13:59:44.363238 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.364867 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:44.367841 7297 consensus_queue.cc:237] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:44.370309 7431 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 13:59:44.370779 7234 consensus_peers.cc:485] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): Couldn't send request to peer 7a6f98c1ec0346ebb30c59ffe8d7c29a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.370954 10752 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:44.373533 9022 raft_consensus.cc:1203] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:44.374410 11716 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.376178 9418 raft_consensus.cc:1261] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 13:59:44.376921 7236 consensus_peers.cc:485] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b0fd771f844fcab38aec069b9abd00. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.377033 10276 consensus_queue.cc:1035] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:44.377013 7431 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.378253 9418 raft_consensus.cc:2932] T a4b0fd771f844fcab38aec069b9abd00 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.379797 7155 catalog_manager.cc:5484] T a4b0fd771f844fcab38aec069b9abd00 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.391666 7297 consensus_queue.cc:237] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } }
I20240503 13:59:44.392668 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:44.393965 7431 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.394605 11876 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.394899 11876 tablet_bootstrap.cc:492] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: Bootstrap complete.
I20240503 13:59:44.395102 11876 ts_tablet_manager.cc:1367] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.000s
I20240503 13:59:44.395207 9152 raft_consensus.cc:1261] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.395622 10385 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.395643 11876 raft_consensus.cc:348] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:44.395758 11876 raft_consensus.cc:729] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5c3042421284f028fa512b7876da405, State: Initialized, Role: LEARNER
I20240503 13:59:44.395851 11876 consensus_queue.cc:260] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
W20240503 13:59:44.396050 7233 consensus_peers.cc:485] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401): Couldn't send request to peer 9d921974718b4613b136eee9d0e1bad2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.396256 11876 ts_tablet_manager.cc:1398] T faa9fb136138439b8f694ac95dbfa2e7 P c5c3042421284f028fa512b7876da405: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:44.396854 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session c5c3042421284f028fa512b7876da405-faa9fb136138439b8f694ac95dbfa2e7 received from {username='slave'} at 127.0.99.135:40203
I20240503 13:59:44.396936 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session c5c3042421284f028fa512b7876da405-faa9fb136138439b8f694ac95dbfa2e7 on tablet faa9fb136138439b8f694ac95dbfa2e7 with peer c5c3042421284f028fa512b7876da405
I20240503 13:59:44.397539 10385 consensus_queue.cc:1035] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.400728 10385 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:44.401358 9152 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:44.405144 7155 catalog_manager.cc:5484] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 6 to 7, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.406484 7297 consensus_queue.cc:237] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:44.409336 9022 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:44.409958 7234 consensus_peers.cc:485] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): Couldn't send request to peer 7a6f98c1ec0346ebb30c59ffe8d7c29a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.410066 10276 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.410377 7561 raft_consensus.cc:1261] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.410773 10276 consensus_queue.cc:1035] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.412962 7431 raft_consensus.cc:2932] T 78bb6d55e8b44732b32da55f00071b21 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } } }
I20240503 13:59:44.416085 9283 raft_consensus.cc:1203] T ade40ddaf0304b1a9e94db6630175c0e P c5c3042421284f028fa512b7876da405 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 13:59:44.417243 10142 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.419427 9022 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.423099 7149 catalog_manager.cc:5484] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 476f53c7e40c4a0b86ffbd818aea818d reported cstate change: config changed from index 6 to 7, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.429004 7561 raft_consensus.cc:2932] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.435500 7690 consensus_queue.cc:237] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:44.437515 7431 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.438129 11855 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:44.440198 9418 raft_consensus.cc:1261] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:44.440647 11717 consensus_queue.cc:1035] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 13:59:44.441854 7626 consensus_peers.cc:485] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881): Couldn't send request to peer e1469ca6a5f945ec89ee1eb8df023b96. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9445bc10fdcf4e9b8d9167ba243bfda5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.442584 11855 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:44.443820 7431 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:44.446471 9418 raft_consensus.cc:2932] T 9445bc10fdcf4e9b8d9167ba243bfda5 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:44.447090 7153 catalog_manager.cc:5484] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 6 to 7, NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } } }
I20240503 13:59:44.469727 12236 ts_tablet_manager.cc:902] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:44.470376 12236 tablet_copy_client.cc:308] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:44.470520 11876 ts_tablet_manager.cc:902] T 3622884b9b314ccca24422d8fa05360a P c5c3042421284f028fa512b7876da405: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
F20240503 13:59:44.470687 11876 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.queue_time_us, tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.queue_time_us) (tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_cpu_time_us, tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_cpu_time_us) (tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_wall_time_us, tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_wall_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.queue_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.queue_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_cpu_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_cpu_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_wall_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_wall_time_us) (tablet-download-pool-3622884b9b314ccca24422d8fa05360a.queue_time_us, tablet-download-pool-3622884b9b314ccca24422d8fa05360a.queue_time_us) (tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.queue_time_us, tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.queue_time_us) (tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_cpu_time_us, tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_cpu_time_us) (tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_wall_time_us, tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_wall_time_us) (tablet-download-pool-491ee8399ef04ae7b93e3228fef05eb0.queue_time_us, tablet-download-pool-491ee8399ef04ae7b93e3228fef05eb0.queue_time_us) (tablet-download-pool-491ee8399ef04ae7b93e3228fef05eb0.run_cpu_time_us, tablet-download-pool-491ee8399ef04ae7b93e3228fef05eb0.run_cpu_time_us) (tablet-download-pool-491ee8399ef04ae7b93e3228fef05eb0.run_wall_time_us, tablet-download-pool-491ee8399ef04ae7b93e3228fef05eb0.run_wall_time_us) (tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.queue_time_us, tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.queue_time_us) (tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_cpu_time_us, tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_cpu_time_us) (tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_wall_time_us, tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_wall_time_us) (tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.queue_time_us, tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.queue_time_us) (tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_cpu_time_us, tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_cpu_time_us) (tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_wall_time_us, tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_wall_time_us) (tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.queue_time_us, tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.queue_time_us) (tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_cpu_time_us, tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_cpu_time_us) (tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_wall_time_us, tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_wall_time_us) (tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.queue_time_us, tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.queue_time_us) (tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.run_cpu_time_us, tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.run_cpu_time_us) (tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.run_wall_time_us, tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.run_wall_time_us) (tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.queue_time_us, tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.queue_time_us) (tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_cpu_time_us, tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_cpu_time_us) (tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_wall_time_us, tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_wall_time_us) (tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.queue_time_us, tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.queue_time_us) (tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_cpu_time_us, tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_cpu_time_us) (tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_wall_time_us, tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_wall_time_us) (tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.queue_time_us, tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.queue_time_us) (tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_cpu_time_us, tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_cpu_time_us) (tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_wall_time_us, tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_wall_time_us) (tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.queue_time_us, tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.queue_time_us) (tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_cpu_time_us, tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_cpu_time_us) (tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_wall_time_us, tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_wall_time_us) (tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.queue_time_us, tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.queue_time_us) (tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_cpu_time_us, tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_cpu_time_us) (tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_wall_time_us, tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_wall_time_us) (tablet-download-pool-fca185e6c04648c5aa4f4ad51b0e1a39.queue_time_us, tablet-download-pool-fca185e6c04648c5aa4f4ad51b0e1a39.queue_time_us) (tablet-download-pool-fca185e6c04648c5aa4f4ad51b0e1a39.run_cpu_time_us, tablet-download-pool-fca185e6c04648c5aa4f4ad51b0e1a39.run_cpu_time_us) (tablet-download-pool-fca185e6c04648c5aa4f4ad51b0e1a39.run_wall_time_us, tablet-download-pool-fca185e6c04648c5aa4f4ad51b0e1a39.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 13:59:44.471155 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 9b04204ad281452794d724fb4d7212f8 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:44957)
I20240503 13:59:44.471302 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 9b04204ad281452794d724fb4d7212f8 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:44957: session id = da390a1a407a4f30b217b3d60fba563f-9b04204ad281452794d724fb4d7212f8
I20240503 13:59:44.472349 7710 tablet_copy_source_session.cc:205] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f222354fdcd google::LogMessage::Fail() at ??:0
I20240503 13:59:44.473835 12236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b04204ad281452794d724fb4d7212f8. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f2223553b93 google::LogMessage::SendToLog() at ??:0
I20240503 13:59:44.476939 12236 tablet_copy_client.cc:783] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:44.477098 12236 tablet_copy_client.cc:647] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:44.478464 12236 tablet_copy_client.cc:523] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:44.480294 12236 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
@ 0x7f222354f7cc google::LogMessage::Flush() at ??:0
@ 0x7f2223550f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f22249b9044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f222499df98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f222499befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f222754e77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 13:59:44.511294 12236 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.511610 12236 tablet_bootstrap.cc:492] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: Bootstrap complete.
I20240503 13:59:44.511869 12236 ts_tablet_manager.cc:1367] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20240503 13:59:44.512421 12236 raft_consensus.cc:348] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.512555 12236 raft_consensus.cc:729] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da390a1a407a4f30b217b3d60fba563f, State: Initialized, Role: LEARNER
I20240503 13:59:44.512653 12236 consensus_queue.cc:260] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.514947 12236 ts_tablet_manager.cc:1398] T 9b04204ad281452794d724fb4d7212f8 P da390a1a407a4f30b217b3d60fba563f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:44.517381 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session da390a1a407a4f30b217b3d60fba563f-9b04204ad281452794d724fb4d7212f8 received from {username='slave'} at 127.0.99.136:44957
I20240503 13:59:44.517472 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session da390a1a407a4f30b217b3d60fba563f-9b04204ad281452794d724fb4d7212f8 on tablet 9b04204ad281452794d724fb4d7212f8 with peer da390a1a407a4f30b217b3d60fba563f
@ 0x7f2227557a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 13:59:44.522051 7297 consensus_queue.cc:237] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:44.524258 7561 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 13:59:44.524755 7234 consensus_peers.cc:485] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): Couldn't send request to peer 7a6f98c1ec0346ebb30c59ffe8d7c29a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
@ 0x7f22275d876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 13:59:44.525457 9418 raft_consensus.cc:1261] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.526437 10752 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 13:59:44.527273 7236 consensus_peers.cc:485] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3622884b9b314ccca24422d8fa05360a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.527526 10276 consensus_queue.cc:1035] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7f22275ec4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 13:59:44.529929 10752 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
@ 0x7f22275d7226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 13:59:44.533407 7155 catalog_manager.cc:5484] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:44.536294 7558 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
I20240503 13:59:44.538918 9418 raft_consensus.cc:2932] T 3622884b9b314ccca24422d8fa05360a P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } } }
@ 0x7f22275e6c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5614371a975c std::function<>::operator()() at ??:0
I20240503 13:59:44.541440 7431 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7"
dest_uuid: "1e5c0b4b872d451dba60797739f83576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41578
I20240503 13:59:44.541622 7431 raft_consensus.cc:595] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:44.545387 12236 ts_tablet_manager.cc:902] T 4582c127b7d34e08a68ea9094de11f19 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:44.545833 12236 tablet_copy_client.cc:308] T 4582c127b7d34e08a68ea9094de11f19 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:44.546358 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 4582c127b7d34e08a68ea9094de11f19 from peer da390a1a407a4f30b217b3d60fba563f ({username='slave'} at 127.0.99.136:44957)
I20240503 13:59:44.546464 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 4582c127b7d34e08a68ea9094de11f19 from peer da390a1a407a4f30b217b3d60fba563f at {username='slave'} at 127.0.99.136:44957: session id = da390a1a407a4f30b217b3d60fba563f-4582c127b7d34e08a68ea9094de11f19
@ 0x7f22249a0e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 13:59:44.547549 7710 tablet_copy_source_session.cc:205] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:44.550107 12236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4582c127b7d34e08a68ea9094de11f19. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f22249a16ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 13:59:44.553540 12236 tablet_copy_client.cc:783] T 4582c127b7d34e08a68ea9094de11f19 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:44.553709 12236 tablet_copy_client.cc:647] T 4582c127b7d34e08a68ea9094de11f19 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Starting download of 1 WAL segments...
@ 0x7f22249a303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5614371a975c std::function<>::operator()() at ??:0
@ 0x7f2224990807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f22251ce6db start_thread at ??:0
@ 0x7f222270671f clone at ??:0
I20240503 13:59:44.566203 12236 tablet_copy_client.cc:523] T 4582c127b7d34e08a68ea9094de11f19 P da390a1a407a4f30b217b3d60fba563f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:44.567814 12236 tablet_bootstrap.cc:492] T 4582c127b7d34e08a68ea9094de11f19 P da390a1a407a4f30b217b3d60fba563f: Bootstrap starting.
I20240503 13:59:44.572472 12343 ts_tablet_manager.cc:902] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:44.573772 12343 tablet_copy_client.cc:308] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:44.574414 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 63552cb8c9c94a07b2a0b157402efa7a from peer 1e5c0b4b872d451dba60797739f83576 ({username='slave'} at 127.0.99.130:48437)
I20240503 13:59:44.574537 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 63552cb8c9c94a07b2a0b157402efa7a from peer 1e5c0b4b872d451dba60797739f83576 at {username='slave'} at 127.0.99.130:48437: session id = 1e5c0b4b872d451dba60797739f83576-63552cb8c9c94a07b2a0b157402efa7a
I20240503 13:59:44.574622 12144 raft_consensus.cc:1050] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 to VOTER
W20240503 13:59:44.574700 12144 raft_consensus.cc:1053] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: Unable to promote non-voter e1469ca6a5f945ec89ee1eb8df023b96: Service unavailable: leader transfer in progress
I20240503 13:59:44.575558 7582 tablet_copy_source_session.cc:205] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:44.576916 12343 ts_tablet_manager.cc:1875] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:44.579682 12343 ts_tablet_manager.cc:1888] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 13:59:44.579883 12343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63552cb8c9c94a07b2a0b157402efa7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:44.601780 12343 tablet_copy_client.cc:783] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:44.601814 12362 ts_tablet_manager.cc:902] T a1115afa5ead4c95aea379ad9987cbe0 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:44.602033 12343 tablet_copy_client.cc:647] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Starting download of 1 WAL segments...
F20240503 13:59:44.602046 12362 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.queue_time_us, tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.queue_time_us) (tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_cpu_time_us, tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_cpu_time_us) (tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_wall_time_us, tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_wall_time_us) (tablet-download-pool-14304180134241c792ec1f23ff89f94c.queue_time_us, tablet-download-pool-14304180134241c792ec1f23ff89f94c.queue_time_us) (tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_cpu_time_us, tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_cpu_time_us) (tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_wall_time_us, tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_wall_time_us) (tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.queue_time_us, tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.queue_time_us) (tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_cpu_time_us, tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_cpu_time_us) (tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_wall_time_us, tablet-download-pool-2212cf72b6ee4ee5b086455a75482e3b.run_wall_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.queue_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.queue_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_cpu_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_cpu_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_wall_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_wall_time_us) (tablet-download-pool-3622884b9b314ccca24422d8fa05360a.queue_time_us, tablet-download-pool-3622884b9b314ccca24422d8fa05360a.queue_time_us) (tablet-download-pool-3622884b9b314ccca24422d8fa05360a.run_cpu_time_us, tablet-download-pool-3622884b9b314ccca24422d8fa05360a.run_cpu_time_us) (tablet-download-pool-3622884b9b314ccca24422d8fa05360a.run_wall_time_us, tablet-download-pool-3622884b9b314ccca24422d8fa05360a.run_wall_time_us) (tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.queue_time_us, tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.queue_time_us) (tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_cpu_time_us, tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_cpu_time_us) (tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_wall_time_us, tablet-download-pool-4582c127b7d34e08a68ea9094de11f19.run_wall_time_us) (tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.queue_time_us, tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.queue_time_us) (tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_cpu_time_us, tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_cpu_time_us) (tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_wall_time_us, tablet-download-pool-902cf3db3581465c83170c4b13b3bffb.run_wall_time_us) (tablet-download-pool-9445bc10fdcf4e9b8d9167ba243bfda5.queue_time_us, tablet-download-pool-9445bc10fdcf4e9b8d9167ba243bfda5.queue_time_us) (tablet-download-pool-9445bc10fdcf4e9b8d9167ba243bfda5.run_cpu_time_us, tablet-download-pool-9445bc10fdcf4e9b8d9167ba243bfda5.run_cpu_time_us) (tablet-download-pool-9445bc10fdcf4e9b8d9167ba243bfda5.run_wall_time_us, tablet-download-pool-9445bc10fdcf4e9b8d9167ba243bfda5.run_wall_time_us) (tablet-download-pool-9b04204ad281452794d724fb4d7212f8.queue_time_us, tablet-download-pool-9b04204ad281452794d724fb4d7212f8.queue_time_us) (tablet-download-pool-9b04204ad281452794d724fb4d7212f8.run_cpu_time_us, tablet-download-pool-9b04204ad281452794d724fb4d7212f8.run_cpu_time_us) (tablet-download-pool-9b04204ad281452794d724fb4d7212f8.run_wall_time_us, tablet-download-pool-9b04204ad281452794d724fb4d7212f8.run_wall_time_us) (tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.queue_time_us, tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.queue_time_us) (tablet-download-pool-a4b0fd771f844fcab38aec069b9abd00.queue_time_us, tablet-download-pool-a4b0fd771f844fcab38aec069b9abd00.queue_time_us) (tablet-download-pool-a4b0fd771f844fcab38aec069b9abd00.run_cpu_time_us, tablet-download-pool-a4b0fd771f844fcab38aec069b9abd00.run_cpu_time_us) (tablet-download-pool-a4b0fd771f844fcab38aec069b9abd00.run_wall_time_us, tablet-download-pool-a4b0fd771f844fcab38aec069b9abd00.run_wall_time_us) (tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.queue_time_us, tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.queue_time_us) (tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_cpu_time_us, tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_cpu_time_us) (tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_wall_time_us, tablet-download-pool-b23f9eeeb4af45b7b679b70514e247b8.run_wall_time_us) (tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.queue_time_us, tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.queue_time_us) (tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_cpu_time_us, tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_cpu_time_us) (tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_wall_time_us, tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_wall_time_us) (tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.queue_time_us, tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.queue_time_us) (tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_cpu_time_us, tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_cpu_time_us) (tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_wall_time_us, tablet-download-pool-faa9fb136138439b8f694ac95dbfa2e7.run_wall_time_us) (tablet-download-pool-ff246d9b2c594282bf2e99d2fd2b617d.queue_time_us, tablet-download-pool-ff246d9b2c594282bf2e99d2fd2b617d.queue_time_us) (tablet-download-pool-ff246d9b2c594282bf2e99d2fd2b617d.run_cpu_time_us, tablet-download-pool-ff246d9b2c594282bf2e99d2fd2b617d.run_cpu_time_us) (tablet-download-pool-ff246d9b2c594282bf2e99d2fd2b617d.run_wall_time_us, tablet-download-pool-ff246d9b2c594282bf2e99d2fd2b617d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 13:59:44.605782 12343 tablet_copy_client.cc:523] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:44.612078 12343 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
@ 0x7fc63fda9dcd google::LogMessage::Fail() at ??:0
@ 0x7fc63fdadb93 google::LogMessage::SendToLog() at ??:0
I20240503 13:59:44.620133 11911 ts_tablet_manager.cc:902] T a1115afa5ead4c95aea379ad9987cbe0 P 476f53c7e40c4a0b86ffbd818aea818d: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
F20240503 13:59:44.620355 11911 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.queue_time_us, tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.queue_time_us) (tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_cpu_time_us, tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_cpu_time_us) (tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_wall_time_us, tablet-download-pool-110ac05b2e8c4594a03225ab2c1ad56d.run_wall_time_us) (tablet-download-pool-14304180134241c792ec1f23ff89f94c.queue_time_us, tablet-download-pool-14304180134241c792ec1f23ff89f94c.queue_time_us) (tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_cpu_time_us, tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_cpu_time_us) (tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_wall_time_us, tablet-download-pool-14304180134241c792ec1f23ff89f94c.run_wall_time_us) (tablet-download-pool-23cf5703ccef47ea927342b9c29f820a.queue_time_us, tablet-download-pool-23cf5703ccef47ea927342b9c29f820a.queue_time_us) (tablet-download-pool-23cf5703ccef47ea927342b9c29f820a.run_cpu_time_us, tablet-download-pool-23cf5703ccef47ea927342b9c29f820a.run_cpu_time_us) (tablet-download-pool-23cf5703ccef47ea927342b9c29f820a.run_wall_time_us, tablet-download-pool-23cf5703ccef47ea927342b9c29f820a.run_wall_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.queue_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.queue_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_cpu_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_cpu_time_us) (tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_wall_time_us, tablet-download-pool-33b7b43742e34431af0c16d320b35dd7.run_wall_time_us) (tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.queue_time_us, tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.queue_time_us) (tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_cpu_time_us, tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_cpu_time_us) (tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_wall_time_us, tablet-download-pool-4e2cabe02dbf4580b53875aaa09aaffa.run_wall_time_us) (tablet-download-pool-86634d81241c44cebcb602a6982f4244.queue_time_us, tablet-download-pool-86634d81241c44cebcb602a6982f4244.queue_time_us) (tablet-download-pool-86634d81241c44cebcb602a6982f4244.run_cpu_time_us, tablet-download-pool-86634d81241c44cebcb602a6982f4244.run_cpu_time_us) (tablet-download-pool-86634d81241c44cebcb602a6982f4244.run_wall_time_us, tablet-download-pool-86634d81241c44cebcb602a6982f4244.run_wall_time_us) (tablet-download-pool-8aff688eaad64d149bdc1c8ab12f4ca1.queue_time_us, tablet-download-pool-8aff688eaad64d149bdc1c8ab12f4ca1.queue_time_us) (tablet-download-pool-8aff688eaad64d149bdc1c8ab12f4ca1.run_cpu_time_us, tablet-download-pool-8aff688eaad64d149bdc1c8ab12f4ca1.run_cpu_time_us) (tablet-download-pool-8aff688eaad64d149bdc1c8ab12f4ca1.run_wall_time_us, tablet-download-pool-8aff688eaad64d149bdc1c8ab12f4ca1.run_wall_time_us) (tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.queue_time_us, tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.queue_time_us) (tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_cpu_time_us, tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_cpu_time_us) (tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_wall_time_us, tablet-download-pool-93d454886dfa40018a4403a3d74b3de2.run_wall_time_us) (tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.queue_time_us, tablet-download-pool-a1115afa5ead4c95aea379ad9987cbe0.queue_time_us) (tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.queue_time_us, tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.queue_time_us) (tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_cpu_time_us, tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_cpu_time_us) (tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_wall_time_us, tablet-download-pool-ade40ddaf0304b1a9e94db6630175c0e.run_wall_time_us) (tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.queue_time_us, tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.queue_time_us) (tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_cpu_time_us, tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_cpu_time_us) (tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_wall_time_us, tablet-download-pool-c1a85f321bfa4faf9b48cb319221fb94.run_wall_time_us) (tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.queue_time_us, tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.queue_time_us) (tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_cpu_time_us, tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_cpu_time_us) (tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_wall_time_us, tablet-download-pool-c74f828c28c64097ab4bf5ec66459d2f.run_wall_time_us) (tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.queue_time_us, tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.queue_time_us) (tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_cpu_time_us, tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_cpu_time_us) (tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_wall_time_us, tablet-download-pool-e0b7095bdf3549fb83b3573a8e65deee.run_wall_time_us) (tablet-download-pool-f3eaa7e5f641436a8f0cfdaf83443ca1.queue_time_us, tablet-download-pool-f3eaa7e5f641436a8f0cfdaf83443ca1.queue_time_us) (tablet-download-pool-f3eaa7e5f641436a8f0cfdaf83443ca1.run_cpu_time_us, tablet-download-pool-f3eaa7e5f641436a8f0cfdaf83443ca1.run_cpu_time_us) (tablet-download-pool-f3eaa7e5f641436a8f0cfdaf83443ca1.run_wall_time_us, tablet-download-pool-f3eaa7e5f641436a8f0cfdaf83443ca1.run_wall_time_us) (tablet-download-pool-f73a5db0cae647d4b3375e6d42358307.queue_time_us, tablet-download-pool-f73a5db0cae647d4b3375e6d42358307.queue_time_us) (tablet-download-pool-f73a5db0cae647d4b3375e6d42358307.run_cpu_time_us, tablet-download-pool-f73a5db0cae647d4b3375e6d42358307.run_cpu_time_us) (tablet-download-pool-f73a5db0cae647d4b3375e6d42358307.run_wall_time_us, tablet-download-pool-f73a5db0cae647d4b3375e6d42358307.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fc63fda97cc google::LogMessage::Flush() at ??:0
@ 0x7fe38c29cdcd google::LogMessage::Fail() at ??:0
I20240503 13:59:44.628839 12382 ts_tablet_manager.cc:902] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
@ 0x7fe38c2a0b93 google::LogMessage::SendToLog() at ??:0
I20240503 13:59:44.629884 12382 tablet_copy_client.cc:308] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44401
I20240503 13:59:44.630962 11716 raft_consensus.cc:1050] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:44.631431 11716 consensus_queue.cc:237] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
@ 0x7fe38c29c7cc google::LogMessage::Flush() at ??:0
I20240503 13:59:44.632153 7558 consensus_queue.cc:237] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
@ 0x7fc63fdaaf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 13:59:44.633829 9152 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.634506 9418 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.634634 9022 raft_consensus.cc:1261] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 13:59:44.635357 7297 raft_consensus.cc:1261] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.635859 10142 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.636269 10142 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
@ 0x7fe38c29df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 13:59:44.637377 10752 consensus_queue.cc:1035] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:44.638569 10197 consensus_queue.cc:1035] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7fc641213044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fc6411f7f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 13:59:44.647836 12070 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.648351 7297 raft_consensus.cc:2932] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.651952 10276 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:44.654733 9418 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:44.654857 9152 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:44.655946 7690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8"
dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33370
I20240503 13:59:44.656122 7690 raft_consensus.cc:595] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:44.656850 7142 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1115afa5ead4c95aea379ad9987cbe0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 13:59:44.657327 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:44.657445 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1115afa5ead4c95aea379ad9987cbe0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:44.658339 7153 catalog_manager.cc:5484] T a1115afa5ead4c95aea379ad9987cbe0 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 7 to 8, VOTER 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } } }
I20240503 13:59:44.661018 7150 catalog_manager.cc:5484] T 14304180134241c792ec1f23ff89f94c P da390a1a407a4f30b217b3d60fba563f reported cstate change: config changed from index 11 to 12, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:44.661355 9022 raft_consensus.cc:2932] T 14304180134241c792ec1f23ff89f94c P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:44.662011 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet a1115afa5ead4c95aea379ad9987cbe0 with delete_type TABLET_DATA_TOMBSTONED (TS 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56594
I20240503 13:59:44.663101 7582 tablet_copy_service.cc:140] P 9d921974718b4613b136eee9d0e1bad2: Received BeginTabletCopySession request for tablet 491ee8399ef04ae7b93e3228fef05eb0 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:45115)
I20240503 13:59:44.663223 7582 tablet_copy_service.cc:161] P 9d921974718b4613b136eee9d0e1bad2: Beginning new tablet copy session on tablet 491ee8399ef04ae7b93e3228fef05eb0 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:45115: session id = 708734e6729441fcb2c5952fa47771d9-491ee8399ef04ae7b93e3228fef05eb0
@ 0x7fc6411f5efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 13:59:44.664167 7582 tablet_copy_source_session.cc:205] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:44.665452 12382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491ee8399ef04ae7b93e3228fef05eb0. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fc643da877a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 13:59:44.668102 12382 tablet_copy_client.cc:783] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:44.668269 12382 tablet_copy_client.cc:647] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:44.669935 12382 tablet_copy_client.cc:523] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:44.671447 12382 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
@ 0x7fe38d706044 kudu::TraceMetrics::InternName() at ??:0
I20240503 13:59:44.678560 12405 tablet_replica.cc:330] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: stopping tablet replica
I20240503 13:59:44.678721 12405 raft_consensus.cc:2229] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:44.678938 12405 raft_consensus.cc:2258] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fe38d6eaf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fc643db1a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe38d6e8efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 13:59:44.689775 12405 ts_tablet_manager.cc:1875] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7fc643e3276f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 13:59:44.689920 11717 raft_consensus.cc:979] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: : Instructing follower 708734e6729441fcb2c5952fa47771d9 to start an election
I20240503 13:59:44.690037 11717 raft_consensus.cc:1067] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Signalling peer 708734e6729441fcb2c5952fa47771d9 to start an election
I20240503 13:59:44.690603 9152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8"
dest_uuid: "708734e6729441fcb2c5952fa47771d9"
from {username='slave'} at 127.0.99.132:40147
I20240503 13:59:44.690791 9152 raft_consensus.cc:482] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:44.690901 9152 raft_consensus.cc:3037] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.692029 9152 raft_consensus.cc:504] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
@ 0x7fe39029b77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 13:59:44.693282 9152 leader_election.cc:288] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
@ 0x7fc643e464f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 13:59:44.693969 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8" candidate_uuid: "708734e6729441fcb2c5952fa47771d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:44.693902 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b04204ad281452794d724fb4d7212f8" candidate_uuid: "708734e6729441fcb2c5952fa47771d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:44.694118 7690 raft_consensus.cc:3032] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:44.694157 7690 raft_consensus.cc:729] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:44.694164 7431 raft_consensus.cc:3037] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.694322 7690 consensus_queue.cc:260] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.694514 7690 raft_consensus.cc:3037] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.694626 12407 ts_tablet_manager.cc:902] T 86634d81241c44cebcb602a6982f4244 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401)
I20240503 13:59:44.695288 7431 raft_consensus.cc:2454] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708734e6729441fcb2c5952fa47771d9 in term 2.
I20240503 13:59:44.695535 7690 raft_consensus.cc:2454] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708734e6729441fcb2c5952fa47771d9 in term 2.
I20240503 13:59:44.695802 9090 leader_election.cc:302] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 708734e6729441fcb2c5952fa47771d9; no voters:
I20240503 13:59:44.696216 11286 raft_consensus.cc:2781] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:44.696421 11286 raft_consensus.cc:686] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [term 2 LEADER]: Becoming Leader. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Running, Role: LEADER
I20240503 13:59:44.696748 11286 consensus_queue.cc:237] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } }
I20240503 13:59:44.700631 7151 catalog_manager.cc:5484] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 reported cstate change: term changed from 1 to 2, leader changed from 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) to 708734e6729441fcb2c5952fa47771d9 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "708734e6729441fcb2c5952fa47771d9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7fe3902a4a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 13:59:44.702980 12426 ts_tablet_manager.cc:902] T 4dffd8718851475099a6696e4a2b8901 P da390a1a407a4f30b217b3d60fba563f: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
@ 0x7fe39032576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe3903394f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe390324226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 13:59:44.716408 12146 raft_consensus.cc:1050] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:44.716925 12146 consensus_queue.cc:237] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:44.718755 9152 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:44.719147 9418 raft_consensus.cc:1261] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Refusing update from remote peer 1e5c0b4b872d451dba60797739f83576: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
@ 0x7fe390333c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 13:59:44.755321 7278 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:44.751991 9398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.785964 10752 raft_consensus.cc:1050] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 1e5c0b4b872d451dba60797739f83576 to VOTER
I20240503 13:59:44.786545 10752 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } }
I20240503 13:59:44.793396 10142 raft_consensus.cc:1050] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96: attempting to promote NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d to VOTER
I20240503 13:59:44.793865 10142 consensus_queue.cc:237] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:44.821033 7297 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14304180134241c792ec1f23ff89f94c"
dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47986
I20240503 13:59:44.821270 7297 raft_consensus.cc:595] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Received request to transfer leadership
@ 0x55d35fe4f75c std::function<>::operator()() at ??:0
I20240503 13:59:44.719574 11197 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7fc643e31226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 13:59:44.850335 12145 consensus_queue.cc:1035] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7fe38d6ede06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc643e40c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 13:59:44.729550 9005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.788486 9022 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 476f53c7e40c4a0b86ffbd818aea818d [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.721683 7409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:44.796238 9024 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.742264 12343 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.796554 9025 raft_consensus.cc:1203] T 93d454886dfa40018a4403a3d74b3de2 P 476f53c7e40c4a0b86ffbd818aea818d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:44.751058 12425 ts_tablet_manager.cc:902] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:44.854723 12343 tablet_bootstrap.cc:492] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Bootstrap complete.
I20240503 13:59:44.854955 12343 ts_tablet_manager.cc:1367] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.243s user 0.029s sys 0.001s
I20240503 13:59:44.855075 12425 tablet_copy_client.cc:308] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:44.855448 12343 raft_consensus.cc:348] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:44.855576 12343 raft_consensus.cc:729] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: LEARNER
@ 0x7fe38d6ee6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 13:59:44.856937 10385 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x555e1e5c175c std::function<>::operator()() at ??:0
@ 0x7fe38d6f003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d35fe4f75c std::function<>::operator()() at ??:0
@ 0x7fe38d6dd807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe38df1b6db start_thread at ??:0
I20240503 13:59:44.855665 12343 consensus_queue.cc:260] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:44.795461 7427 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.789831 7431 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.886358 12343 ts_tablet_manager.cc:1398] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 13:59:44.867808 12145 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
@ 0x7fe38b45371f clone at ??:0
I20240503 13:59:44.890175 9418 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
@ 0x7fc6411fae06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 13:59:44.937300 12146 raft_consensus.cc:979] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576: : Instructing follower da390a1a407a4f30b217b3d60fba563f to start an election
I20240503 13:59:44.951066 12146 raft_consensus.cc:1067] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Signalling peer da390a1a407a4f30b217b3d60fba563f to start an election
W20240503 13:59:44.951238 7367 connection.cc:535] client connection to 127.0.99.135:41051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:59:44.951365 7367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7fc6411fb6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 13:59:44.954401 9418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7"
dest_uuid: "da390a1a407a4f30b217b3d60fba563f"
from {username='slave'} at 127.0.99.130:41407
I20240503 13:59:44.954607 9418 raft_consensus.cc:482] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:44.954727 9418 raft_consensus.cc:3037] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.955819 9418 raft_consensus.cc:504] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:44.957024 9418 leader_election.cc:288] T 33b7b43742e34431af0c16d320b35dd7 P da390a1a407a4f30b217b3d60fba563f [CANDIDATE]: Term 2 election: Requested vote from peers 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), c5c3042421284f028fa512b7876da405 (127.0.99.135:41051)
I20240503 13:59:44.957691 7425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b7b43742e34431af0c16d320b35dd7" candidate_uuid: "da390a1a407a4f30b217b3d60fba563f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: true dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:44.957861 7425 raft_consensus.cc:3032] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:44.957903 7425 raft_consensus.cc:729] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
@ 0x7fc6411fd03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555e1e5c175c std::function<>::operator()() at ??:0
W20240503 13:59:44.965533 9355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
@ 0x7fc6411ea807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc641a286db start_thread at ??:0
@ 0x7fc63ef6071f clone at ??:0
I20240503 13:59:44.857389 10385 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:44.732841 9136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:44.870992 12552 ts_tablet_manager.cc:902] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: Initiating tablet copy from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133)
I20240503 13:59:44.982919 12552 tablet_copy_client.cc:308] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43133
I20240503 13:59:44.983137 11716 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:44.765235 12464 ts_tablet_manager.cc:902] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Initiating tablet copy from peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237)
I20240503 13:59:44.984263 12464 tablet_copy_client.cc:308] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46237
I20240503 13:59:44.889966 10752 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:44.765828 12382 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:44.985487 12382 tablet_bootstrap.cc:492] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:44.985719 12382 ts_tablet_manager.cc:1367] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.314s user 0.029s sys 0.000s
I20240503 13:59:44.797430 9152 raft_consensus.cc:1261] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.868642 9151 raft_consensus.cc:2932] T 902cf3db3581465c83170c4b13b3bffb P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
W20240503 13:59:44.950207 7236 connection.cc:535] client connection to 127.0.99.135:41051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:44.993021 12382 raft_consensus.cc:348] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:45.007297 12382 raft_consensus.cc:729] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:45.007508 12382 consensus_queue.cc:260] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:45.007923 12382 ts_tablet_manager.cc:1398] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:59:44.891463 7151 catalog_manager.cc:5484] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 reported cstate change: config changed from index 10 to 11, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:45.087509 9136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 13:59:44.990432 7236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:59:45.150100 7236 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.0.99.135:41051, user_credentials={real_user=slave}} blocked reactor thread for 159693us
I20240503 13:59:44.958112 7425 consensus_queue.cc:260] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:45.150477 7425 raft_consensus.cc:3037] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.990923 7297 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a"
dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48026
I20240503 13:59:45.151404 11716 consensus_queue.cc:1035] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.151741 7425 raft_consensus.cc:2454] T 33b7b43742e34431af0c16d320b35dd7 P 1e5c0b4b872d451dba60797739f83576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da390a1a407a4f30b217b3d60fba563f in term 2.
I20240503 13:59:44.736116 7672 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:44.991413 10752 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:44.744912 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:44.788296 7558 raft_consensus.cc:1261] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer e1469ca6a5f945ec89ee1eb8df023b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:44.988806 7450 tablet_copy_service.cc:140] P 1e5c0b4b872d451dba60797739f83576: Received BeginTabletCopySession request for tablet fca185e6c04648c5aa4f4ad51b0e1a39 from peer 708734e6729441fcb2c5952fa47771d9 ({username='slave'} at 127.0.99.134:47471)
I20240503 13:59:45.152292 7450 tablet_copy_service.cc:161] P 1e5c0b4b872d451dba60797739f83576: Beginning new tablet copy session on tablet fca185e6c04648c5aa4f4ad51b0e1a39 from peer 708734e6729441fcb2c5952fa47771d9 at {username='slave'} at 127.0.99.134:47471: session id = 708734e6729441fcb2c5952fa47771d9-fca185e6c04648c5aa4f4ad51b0e1a39
I20240503 13:59:45.153273 7450 tablet_copy_source_session.cc:205] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:44.738364 12405 ts_tablet_manager.cc:1888] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 13:59:44.752866 7690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d"
dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33398
I20240503 13:59:45.153707 12405 log.cc:1196] T a1115afa5ead4c95aea379ad9987cbe0 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1115afa5ead4c95aea379ad9987cbe0
I20240503 13:59:45.153743 7690 raft_consensus.cc:595] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:44.887951 7582 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 1e5c0b4b872d451dba60797739f83576-63552cb8c9c94a07b2a0b157402efa7a received from {username='slave'} at 127.0.99.130:48437
I20240503 13:59:45.153864 7582 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 1e5c0b4b872d451dba60797739f83576-63552cb8c9c94a07b2a0b157402efa7a on tablet 63552cb8c9c94a07b2a0b157402efa7a with peer 1e5c0b4b872d451dba60797739f83576
I20240503 13:59:44.992159 7427 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:45.154506 7581 tablet_copy_service.cc:342] P 9d921974718b4613b136eee9d0e1bad2: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-491ee8399ef04ae7b93e3228fef05eb0 received from {username='slave'} at 127.0.99.134:45115
I20240503 13:59:45.154580 7581 tablet_copy_service.cc:433] P 9d921974718b4613b136eee9d0e1bad2: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-491ee8399ef04ae7b93e3228fef05eb0 on tablet 491ee8399ef04ae7b93e3228fef05eb0 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:45.154830 10385 consensus_queue.cc:1035] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:44.848747 11717 raft_consensus.cc:1050] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: attempting to promote NON_VOTER e1469ca6a5f945ec89ee1eb8df023b96 to VOTER
I20240503 13:59:45.155977 11717 consensus_queue.cc:237] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: false } }
I20240503 13:59:45.156857 7297 raft_consensus.cc:595] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:45.039556 7431 raft_consensus.cc:1203] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 13:59:44.951112 7498 connection.cc:535] client connection to 127.0.99.135:41051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:59:45.157353 7498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:44.856434 7710 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet e0b7095bdf3549fb83b3573a8e65deee from peer 1e5c0b4b872d451dba60797739f83576 ({username='slave'} at 127.0.99.130:58063)
I20240503 13:59:45.157590 7710 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet e0b7095bdf3549fb83b3573a8e65deee from peer 1e5c0b4b872d451dba60797739f83576 at {username='slave'} at 127.0.99.130:58063: session id = 1e5c0b4b872d451dba60797739f83576-e0b7095bdf3549fb83b3573a8e65deee
I20240503 13:59:45.157692 12464 ts_tablet_manager.cc:1875] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:45.157672 7561 raft_consensus.cc:2932] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:45.080227 7423 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 1e5c0b4b872d451dba60797739f83576 [term 2 FOLLOWER]: Refusing update from remote peer 708734e6729441fcb2c5952fa47771d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 13:59:44.864265 12553 ts_tablet_manager.cc:902] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:44.992882 12649 ts_tablet_manager.cc:902] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:45.159164 12553 tablet_copy_client.cc:308] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
W20240503 13:59:44.965867 7367 consensus_peers.cc:485] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.159442 7367 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:41051, user_credentials={real_user=slave}} blocked reactor thread for 193579us
W20240503 13:59:45.159639 7367 consensus_peers.cc:485] T fca185e6c04648c5aa4f4ad51b0e1a39 P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.159744 7367 consensus_peers.cc:485] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.159806 7367 consensus_peers.cc:485] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.263087 9471 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 13:59:37.802506 9474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.802788 9474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.808770 9474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:38.880249 9474 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 13:59:39.423947 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.424122 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.424161 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.424252 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.424290 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.424341 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.424388 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.424424 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.424458 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.424491 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.424537 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.424583 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.424621 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.424672 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.424707 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:39.461796 9474 rebalancer_tool.cc:1074] tablet 491ee8399ef04ae7b93e3228fef05eb0: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.506320 9474 rebalancer_tool.cc:1074] tablet 9445bc10fdcf4e9b8d9167ba243bfda5: '9d921974718b4613b136eee9d0e1bad2' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.562646 9474 rebalancer_tool.cc:1074] tablet 8aff688eaad64d149bdc1c8ab12f4ca1: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:39.673637 9474 rebalancer_tool.cc:1074] tablet c74f828c28c64097ab4bf5ec66459d2f: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:39.696097 9474 rebalancer_tool.cc:1074] tablet 33b7b43742e34431af0c16d320b35dd7: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.724443 9474 rebalancer_tool.cc:1084] tablet 2212cf72b6ee4ee5b086455a75482e3b: '9d921974718b4613b136eee9d0e1bad2' -> 'da390a1a407a4f30b217b3d60fba563f' move ignored: Illegal state: tablet 2212cf72b6ee4ee5b086455a75482e3b: replica da390a1a407a4f30b217b3d60fba563f is already present
I20240503 13:59:39.724560 9474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:40.317421 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.317744 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.318117 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.319073 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.319244 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.319312 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.319527 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.319589 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.319648 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.319774 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.319954 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.510170 9474 rebalancer_tool.cc:1084] tablet 86634d81241c44cebcb602a6982f4244: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 86634d81241c44cebcb602a6982f4244: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:40.510275 9474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:41.921730 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.921872 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.921948 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.922076 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.922158 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.922217 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.922513 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.922595 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:41.922639 9474 rebalancer_tool.cc:1205] No I20240503 13:59:44.882041 12563 ts_tablet_manager.cc:902] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
I20240503 13:59:45.161854 12649 tablet_copy_client.cc:308] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:45.161078 10142 raft_consensus.cc:979] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96: : Instructing follower 1e5c0b4b872d451dba60797739f83576 to start an election
I20240503 13:59:45.160148 12464 ts_tablet_manager.cc:1888] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 13:59:45.322306 10142 raft_consensus.cc:1067] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Signalling peer 1e5c0b4b872d451dba60797739f83576 to start an election
I20240503 13:59:45.322422 12464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca185e6c04648c5aa4f4ad51b0e1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.322698 12563 tablet_copy_client.cc:308] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
W20240503 13:59:44.954064 7629 connection.cc:535] client connection to 127.0.99.135:41051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:44.987208 7713 tablet_copy_service.cc:140] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Received BeginTabletCopySession request for tablet 9445bc10fdcf4e9b8d9167ba243bfda5 from peer e1469ca6a5f945ec89ee1eb8df023b96 ({username='slave'} at 127.0.99.129:43167)
W20240503 13:59:45.323005 7629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:45.323065 7713 tablet_copy_service.cc:161] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Beginning new tablet copy session on tablet 9445bc10fdcf4e9b8d9167ba243bfda5 from peer e1469ca6a5f945ec89ee1eb8df023b96 at {username='slave'} at 127.0.99.129:43167: session id = e1469ca6a5f945ec89ee1eb8df023b96-9445bc10fdcf4e9b8d9167ba243bfda5
I20240503 13:59:45.323195 7319 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 78bb6d55e8b44732b32da55f00071b21 from peer 9d921974718b4613b136eee9d0e1bad2 ({username='slave'} at 127.0.99.131:48917)
I20240503 13:59:45.323311 7319 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 78bb6d55e8b44732b32da55f00071b21 from peer 9d921974718b4613b136eee9d0e1bad2 at {username='slave'} at 127.0.99.131:48917: session id = 9d921974718b4613b136eee9d0e1bad2-78bb6d55e8b44732b32da55f00071b21
W20240503 13:59:45.323944 7629 connection.cc:698] client connection to 127.0.99.136:43485 send error: Network error: sendmsg error: Broken pipe (error 32)
I20240503 13:59:45.324092 12464 tablet_copy_client.cc:783] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:45.324200 7713 tablet_copy_source_session.cc:205] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:45.324275 12464 tablet_copy_client.cc:647] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:45.170377 7561 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:45.324379 7319 tablet_copy_source_session.cc:205] T 78bb6d55e8b44732b32da55f00071b21 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:45.168318 11716 raft_consensus.cc:979] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96: : Instructing follower 708734e6729441fcb2c5952fa47771d9 to start an election
I20240503 13:59:45.324671 11716 raft_consensus.cc:1067] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Signalling peer 708734e6729441fcb2c5952fa47771d9 to start an election
W20240503 13:59:45.197368 7498 consensus_peers.cc:485] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.922677 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.922713 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.922762 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.922806 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:42.062741 9474 rebalancer_tool.cc:1074] tablet ade40ddaf0304b1a9e94db6630175c0e: '9d921974718b4613b136eee9d0e1bad2' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:42.250116 9474 rebalancer_tool.cc:1084] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '1e5c0b4b872d451dba60797739f83576' -> 'da390a1a407a4f30b217b3d60fba563f' move ignored: Illegal state: tablet 110ac05b2e8c4594a03225ab2c1ad56d: replica da390a1a407a4f30b217b3d60fba563f is already present
I20240503 13:59:42.250221 9474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.031123 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server e1469ca6a5f945ec89ee1eb8df023b96
W20240503 13:59:43.031276 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:43.031389 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.031468 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:43.031539 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server da390a1a407a4f30b217b3d60fba563f to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:43.098457 9474 rebalancer_tool.cc:1074] tablet 14304180134241c792ec1f23ff89f94c: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '' move scheduled
I20240503 13:59:43.237558 9474 rebalancer_tool.cc:1084] tablet ff246d9b2c594282bf2e99d2fd2b617d: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move ignored: Illegal state: tablet ff246d9b2c594282bf2e99d2fd2b617d: replica 708734e6729441fcb2c5952fa47771d9 is already present
I20240503 13:59:43.237668 9474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.811601 9474 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server e1469ca6a5f945ec89ee1eb8df023b96
I20240503 13:59:43.909072 9474 rebalancer_tool.cc:1074] tablet faa9fb136138439b8f694ac95dbfa2e7: '708734e6729441fcb2c5952fa47771d9' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:43.932891 9474 rebalancer_tool.cc:1074] tablet 4582c127b7d34e08a68ea9094de11f19: '1e5c0b4b872d451dba60797739f83576' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:43.968019 9474 rebalancer_tool.cc:1074] tablet ff246d9b2c594282bf2e99d2fd2b617d: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '9d921974718b4613b136eee9d0e1bad2' move scheduled
I20240503 13:59:44.064201 9474 rebalancer_tool.cc:1074] tablet a1115afa5ead4c95aea379ad9987cbe0: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:44.224218 9474 rebalancer_tool.cc:1074] tablet fca185e6c04648c5aa4f4ad51b0e1a39: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:44.261744 9474 rebalancer_tool.cc:1074] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '9d921974718b4613b136eee9d0e1bad2' -> '1e5c0b4b872d451dba60797739f83576' move scheduled
I20240503 13:59:44.403762 9474 rebalancer_tool.cc:1074] tablet 78bb6d55e8b44732b32da55f00071b21: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '9d921974718b4613b136eee9d0e1bad2' move scheduled
I20240503 13:59:44.432691 9474 rebalancer_tool.cc:1084] tablet a4b0fd771f844fcab38aec069b9abd00: '1e5c0b4b872d451dba60797739f83576' -> '7a6f98c1ec0346ebb30c59ffe8d7c29a' move ignored: Illegal state: tablet a4b0fd771f844fcab38aec069b9abd00: replica 7a6f98c1ec0346ebb30c59ffe8d7c29a is already present
I20240503 13:59:44.432806 9474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:44.952358 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:59:44.737340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:41051 (local address 127.0.0.1:45994)
0503 13:59:44.738271 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0503 13:59:44.738284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 13:59:44.738358 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 13:59:44.951513 (+213155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":766}
Illegal state: tablet server c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): unacceptable health status UNAVAILABLE
I20240503 13:59:45.168454 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet f3eaa7e5f641436a8f0cfdaf83443ca1 from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a ({username='slave'} at 127.0.99.132:52805)
W20240503 13:59:45.362514 7498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:41051, user_credentials={real_user=slave}} blocked reactor thread for 165136us
I20240503 13:59:45.362664 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet f3eaa7e5f641436a8f0cfdaf83443ca1 from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a at {username='slave'} at 127.0.99.132:52805: session id = 7a6f98c1ec0346ebb30c59ffe8d7c29a-f3eaa7e5f641436a8f0cfdaf83443ca1
I20240503 13:59:45.170982 7153 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:45.168674 7297 raft_consensus.cc:1261] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Refusing update from remote peer 7a6f98c1ec0346ebb30c59ffe8d7c29a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 13:59:45.362848 7498 consensus_peers.cc:485] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.362900 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.326804 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9445bc10fdcf4e9b8d9167ba243bfda5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 13:59:45.363127 7498 connection.cc:698] client connection to 127.0.99.136:43485 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240503 13:59:45.054610 12660 ts_tablet_manager.cc:902] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Initiating tablet copy from peer e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881)
W20240503 13:59:45.159840 7367 consensus_peers.cc:462] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
W20240503 13:59:45.363499 7498 connection.cc:698] client connection to 127.0.99.133:36553 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240503 13:59:45.363590 7367 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.0.99.135:41051, user_credentials={real_user=slave}} blocked reactor thread for 203724us
I20240503 13:59:45.364104 12660 tablet_copy_client.cc:308] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41881
I20240503 13:59:45.325883 12649 ts_tablet_manager.cc:1875] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 13:59:45.364491 7367 connection.cc:698] client connection to 127.0.99.136:43485 send error: Network error: sendmsg error: Broken pipe (error 32)
I20240503 13:59:45.162892 12424 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.365873 12552 tablet_copy_client.cc:783] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Starting download of 0 data blocks...
W20240503 13:59:45.170948 7236 consensus_peers.cc:462] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
W20240503 13:59:45.366029 7236 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.0.99.135:41051, user_credentials={real_user=slave}} blocked reactor thread for 195073us
W20240503 13:59:45.366098 7236 consensus_peers.cc:462] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
W20240503 13:59:45.366304 7236 connection.cc:698] client connection to 127.0.99.133:36553 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240503 13:59:45.366588 7236 connection.cc:698] client connection to 127.0.99.136:43485 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240503 13:59:45.089435 7673 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:45.366683 12649 ts_tablet_manager.cc:1888] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 13:59:45.163868 7710 tablet_copy_source_session.cc:205] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 13:59:45.328220 7629 consensus_peers.cc:485] T 9445bc10fdcf4e9b8d9167ba243bfda5 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.366897 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bb6d55e8b44732b32da55f00071b21. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 13:59:45.366969 7629 consensus_peers.cc:485] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.367029 7629 consensus_peers.cc:485] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.295433 7144 connection.cc:535] server connection from 127.0.99.136:53829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:45.368011 7320 tablet_copy_source_session.cc:205] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 13:59:45.368077 7629 consensus_peers.cc:485] T 4582c127b7d34e08a68ea9094de11f19 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.368139 7629 consensus_peers.cc:485] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.368194 7629 consensus_peers.cc:485] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.179538 7409 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 13:59:45.369602 7498 consensus_peers.cc:485] T ade40ddaf0304b1a9e94db6630175c0e P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.369659 7498 consensus_peers.cc:462] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111)
W20240503 13:59:45.369688 7498 consensus_peers.cc:485] T 8aff688eaad64d149bdc1c8ab12f4ca1 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.369853 7498 consensus_peers.cc:462] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
W20240503 13:59:45.369895 7498 consensus_peers.cc:485] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.369928 7498 consensus_peers.cc:462] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
I20240503 13:59:45.369793 10142 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:45.164305 9151 raft_consensus.cc:1203] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:45.325170 7143 catalog_manager.cc:4830] TS 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133): tablet a1115afa5ead4c95aea379ad9987cbe0 (table rebalance_test_table_3 [id=3986eba04fca4c2fbdbe763bfcab3d02]) successfully deleted
W20240503 13:59:45.370489 7143 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.99.132:43133, user_credentials={real_user=slave}} blocked reactor thread for 45328.5us
I20240503 13:59:45.328334 10291 raft_consensus.cc:979] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a: : Instructing follower 708734e6729441fcb2c5952fa47771d9 to start an election
I20240503 13:59:45.371425 10291 raft_consensus.cc:1067] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Signalling peer 708734e6729441fcb2c5952fa47771d9 to start an election
I20240503 13:59:45.371425 9154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14304180134241c792ec1f23ff89f94c"
dest_uuid: "708734e6729441fcb2c5952fa47771d9"
from {username='slave'} at 127.0.99.129:55691
I20240503 13:59:45.371546 9154 raft_consensus.cc:482] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:45.371598 9154 raft_consensus.cc:3037] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.210866 7410 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:45.372625 9154 raft_consensus.cc:504] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
W20240503 13:59:45.295043 7364 connection.cc:535] server connection from 127.0.99.136:40713 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:45.373204 9154 leader_election.cc:288] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: Requested vote from peers e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485), 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553)
I20240503 13:59:45.325393 7423 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a"
dest_uuid: "1e5c0b4b872d451dba60797739f83576"
from {username='slave'} at 127.0.99.129:39677
I20240503 13:59:45.373518 7423 raft_consensus.cc:482] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:45.373595 7423 raft_consensus.cc:3037] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:45.296319 12695 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:59:45.100658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:43485 (local address 127.0.99.134:45745)
0503 13:59:45.100895 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0503 13:59:45.100910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 13:59:45.101009 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 13:59:45.295679 (+194670us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20240503 13:59:45.374214 12649 tablet_copy_client.cc:783] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:45.374394 12649 tablet_copy_client.cc:647] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: tablet copy: Starting download of 1 WAL segments...
W20240503 13:59:45.374513 9092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 13:59:45.374689 7423 raft_consensus.cc:504] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } }
I20240503 13:59:45.375075 7692 raft_consensus.cc:1261] T 9b04204ad281452794d724fb4d7212f8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 2 FOLLOWER]: Refusing update from remote peer 708734e6729441fcb2c5952fa47771d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 13:59:45.375116 7423 leader_election.cc:288] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 2 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129:41881), 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553)
I20240503 13:59:45.376454 12425 ts_tablet_manager.cc:1875] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:45.376869 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:45.376981 7561 raft_consensus.cc:3037] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.376972 12553 ts_tablet_manager.cc:1875] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:45.378089 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cf5703ccef47ea927342b9c29f820a" candidate_uuid: "1e5c0b4b872d451dba60797739f83576" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:45.378212 7297 raft_consensus.cc:3032] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:45.378270 7297 raft_consensus.cc:729] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:45.378436 7297 consensus_queue.cc:260] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } }
I20240503 13:59:45.378646 7297 raft_consensus.cc:3037] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.377933 7561 raft_consensus.cc:2454] T 23cf5703ccef47ea927342b9c29f820a P 9d921974718b4613b136eee9d0e1bad2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 2.
I20240503 13:59:45.378866 12425 ts_tablet_manager.cc:1888] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 13:59:45.379037 12425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b7095bdf3549fb83b3573a8e65deee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.379252 12553 ts_tablet_manager.cc:1888] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 13:59:45.379415 12553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3eaa7e5f641436a8f0cfdaf83443ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.379673 7297 raft_consensus.cc:2454] T 23cf5703ccef47ea927342b9c29f820a P e1469ca6a5f945ec89ee1eb8df023b96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5c0b4b872d451dba60797739f83576 in term 2.
I20240503 13:59:45.379999 7364 leader_election.cc:302] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters:
I20240503 13:59:45.380113 12425 tablet_copy_client.cc:783] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:45.380240 12425 tablet_copy_client.cc:647] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:45.380337 11197 raft_consensus.cc:2781] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:45.380409 11197 raft_consensus.cc:686] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [term 2 LEADER]: Becoming Leader. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Running, Role: LEADER
I20240503 13:59:45.380532 11197 consensus_queue.cc:237] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } }
I20240503 13:59:45.380625 12553 tablet_copy_client.cc:783] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:45.380770 12553 tablet_copy_client.cc:647] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:45.380862 10082 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:45.381196 10082 consensus_queue.cc:1035] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:45.383291 12424 consensus_queue.cc:1035] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.386909 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet a4b0fd771f844fcab38aec069b9abd00 from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a ({username='slave'} at 127.0.99.132:52805)
I20240503 13:59:45.387037 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet a4b0fd771f844fcab38aec069b9abd00 from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a at {username='slave'} at 127.0.99.132:52805: session id = 7a6f98c1ec0346ebb30c59ffe8d7c29a-a4b0fd771f844fcab38aec069b9abd00
I20240503 13:59:45.387658 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:45.388161 7320 tablet_copy_source_session.cc:205] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 13:59:45.388890 7236 consensus_peers.cc:485] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.388960 7236 consensus_peers.cc:485] T f3eaa7e5f641436a8f0cfdaf83443ca1 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.389001 7236 consensus_peers.cc:485] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.389861 7673 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:45.390815 9152 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
W20240503 13:59:45.391731 7498 consensus_peers.cc:485] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.391844 7498 consensus_peers.cc:485] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.392784 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14304180134241c792ec1f23ff89f94c" candidate_uuid: "708734e6729441fcb2c5952fa47771d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
I20240503 13:59:45.392920 7300 raft_consensus.cc:3032] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:45.392964 7300 raft_consensus.cc:729] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Running, Role: LEADER
I20240503 13:59:45.393133 7300 consensus_queue.cc:260] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } }
I20240503 13:59:45.393352 7300 raft_consensus.cc:3037] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.394093 9154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d"
dest_uuid: "708734e6729441fcb2c5952fa47771d9"
from {username='slave'} at 127.0.99.132:40147
I20240503 13:59:45.394243 9154 raft_consensus.cc:482] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:45.394302 9154 raft_consensus.cc:3037] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.394412 12649 tablet_copy_client.cc:523] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.394418 7300 raft_consensus.cc:2454] T 14304180134241c792ec1f23ff89f94c P e1469ca6a5f945ec89ee1eb8df023b96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708734e6729441fcb2c5952fa47771d9 in term 2.
I20240503 13:59:45.395361 9154 raft_consensus.cc:504] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:45.395495 7425 raft_consensus.cc:2932] T f73a5db0cae647d4b3375e6d42358307 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } }
I20240503 13:59:45.395996 9154 leader_election.cc:288] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: Requested vote from peers 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132:43133), 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237), da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485)
I20240503 13:59:45.397670 7425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d" candidate_uuid: "708734e6729441fcb2c5952fa47771d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "1e5c0b4b872d451dba60797739f83576"
I20240503 13:59:45.397795 7425 raft_consensus.cc:3037] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.398743 7425 raft_consensus.cc:2454] T ff246d9b2c594282bf2e99d2fd2b617d P 1e5c0b4b872d451dba60797739f83576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708734e6729441fcb2c5952fa47771d9 in term 2.
I20240503 13:59:45.328333 12464 tablet_copy_client.cc:523] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.399545 12563 ts_tablet_manager.cc:1875] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:45.400676 9136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:45.401237 12464 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Bootstrap starting.
I20240503 13:59:45.401729 12563 ts_tablet_manager.cc:1888] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 13:59:45.401885 12563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0fd771f844fcab38aec069b9abd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.402928 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff246d9b2c594282bf2e99d2fd2b617d" candidate_uuid: "708734e6729441fcb2c5952fa47771d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
I20240503 13:59:45.403076 7692 raft_consensus.cc:3032] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:45.403175 7692 raft_consensus.cc:729] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Running, Role: LEADER
I20240503 13:59:45.403316 7692 consensus_queue.cc:260] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:45.403512 7692 raft_consensus.cc:3037] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.404440 7692 raft_consensus.cc:2454] T ff246d9b2c594282bf2e99d2fd2b617d P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708734e6729441fcb2c5952fa47771d9 in term 2.
I20240503 13:59:45.404889 12552 tablet_copy_client.cc:647] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Starting download of 1 WAL segments...
W20240503 13:59:45.405135 7236 consensus_peers.cc:485] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.405892 7278 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:45.408056 7153 catalog_manager.cc:5484] T f73a5db0cae647d4b3375e6d42358307 P e1469ca6a5f945ec89ee1eb8df023b96 reported cstate change: config changed from index 7 to 8, 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:45.410063 7320 tablet_copy_service.cc:140] P e1469ca6a5f945ec89ee1eb8df023b96: Received BeginTabletCopySession request for tablet 3622884b9b314ccca24422d8fa05360a from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a ({username='slave'} at 127.0.99.132:52805)
I20240503 13:59:45.410178 7320 tablet_copy_service.cc:161] P e1469ca6a5f945ec89ee1eb8df023b96: Beginning new tablet copy session on tablet 3622884b9b314ccca24422d8fa05360a from peer 7a6f98c1ec0346ebb30c59ffe8d7c29a at {username='slave'} at 127.0.99.132:52805: session id = 7a6f98c1ec0346ebb30c59ffe8d7c29a-3622884b9b314ccca24422d8fa05360a
I20240503 13:59:45.411132 7320 tablet_copy_source_session.cc:205] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:45.411211 9090 leader_election.cc:302] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1e5c0b4b872d451dba60797739f83576, 708734e6729441fcb2c5952fa47771d9, 7a6f98c1ec0346ebb30c59ffe8d7c29a; no voters:
I20240503 13:59:45.411420 12424 raft_consensus.cc:2781] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:45.402943 12563 tablet_copy_client.cc:783] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:45.411482 12424 raft_consensus.cc:686] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [term 2 LEADER]: Becoming Leader. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Running, Role: LEADER
I20240503 13:59:45.411641 12563 tablet_copy_client.cc:647] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:45.411621 12424 consensus_queue.cc:237] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
W20240503 13:59:45.414566 7367 consensus_peers.cc:462] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
W20240503 13:59:45.414652 7367 consensus_peers.cc:485] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.414700 7367 consensus_peers.cc:485] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.415004 7236 consensus_peers.cc:485] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.415197 7367 consensus_peers.cc:485] T 4e2cabe02dbf4580b53875aaa09aaffa P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.415593 7236 consensus_peers.cc:462] T a4b0fd771f844fcab38aec069b9abd00 P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
I20240503 13:59:45.416783 12552 tablet_copy_client.cc:523] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.417037 12660 ts_tablet_manager.cc:1875] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 13:59:45.418884 12425 tablet_copy_client.cc:523] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.419453 12552 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap starting.
I20240503 13:59:45.419574 12660 ts_tablet_manager.cc:1888] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 13:59:45.419772 12660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3622884b9b314ccca24422d8fa05360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.420440 12425 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Bootstrap starting.
W20240503 13:59:45.420656 7629 consensus_peers.cc:485] T 93d454886dfa40018a4403a3d74b3de2 P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.420948 12660 tablet_copy_client.cc:783] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:45.421092 12660 tablet_copy_client.cc:647] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:45.422863 12649 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap starting.
I20240503 13:59:45.424098 7151 catalog_manager.cc:5484] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 reported cstate change: term changed from 1 to 2, leader changed from 7a6f98c1ec0346ebb30c59ffe8d7c29a (127.0.99.132) to 708734e6729441fcb2c5952fa47771d9 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "708734e6729441fcb2c5952fa47771d9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:45.424636 7542 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:45.428498 12563 tablet_copy_client.cc:523] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.428576 7278 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:45.430594 12563 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
W20240503 13:59:45.440392 7367 consensus_peers.cc:462] T 2212cf72b6ee4ee5b086455a75482e3b P 1e5c0b4b872d451dba60797739f83576 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
W20240503 13:59:45.441015 7367 consensus_peers.cc:485] T c1a85f321bfa4faf9b48cb319221fb94 P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.441100 7367 leader_election.cc:334] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111)
W20240503 13:59:45.441241 7367 consensus_peers.cc:485] T c74f828c28c64097ab4bf5ec66459d2f P 1e5c0b4b872d451dba60797739f83576 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.441882 12553 tablet_copy_client.cc:523] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.442647 7149 catalog_manager.cc:5484] T 23cf5703ccef47ea927342b9c29f820a P 1e5c0b4b872d451dba60797739f83576 reported cstate change: term changed from 1 to 2, leader changed from e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) to 1e5c0b4b872d451dba60797739f83576 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "1e5c0b4b872d451dba60797739f83576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:45.444386 12553 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
W20240503 13:59:45.452051 7629 consensus_peers.cc:485] T e0b7095bdf3549fb83b3573a8e65deee P 7a6f98c1ec0346ebb30c59ffe8d7c29a -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.457396 12711 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: false } } }
I20240503 13:59:45.459617 7561 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: false } } }
W20240503 13:59:45.461273 7367 consensus_peers.cc:485] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.461625 7297 raft_consensus.cc:2932] T b23f9eeeb4af45b7b679b70514e247b8 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: false } } }
I20240503 13:59:45.462144 7149 catalog_manager.cc:5484] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a reported cstate change: config changed from index 10 to 11, e1469ca6a5f945ec89ee1eb8df023b96 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:45.466162 9092 leader_election.cc:334] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
W20240503 13:59:45.466360 9092 leader_election.cc:334] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
W20240503 13:59:45.466449 9092 consensus_peers.cc:485] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:45.466536 9092 leader_election.cc:334] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111)
I20240503 13:59:45.466578 9092 leader_election.cc:302] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 708734e6729441fcb2c5952fa47771d9, e1469ca6a5f945ec89ee1eb8df023b96; no voters: 476f53c7e40c4a0b86ffbd818aea818d, da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:45.466817 12660 tablet_copy_client.cc:523] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:45.467280 11286 raft_consensus.cc:2726] T 14304180134241c792ec1f23ff89f94c P 708734e6729441fcb2c5952fa47771d9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 13:59:45.468637 12660 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap starting.
I20240503 13:59:45.521088 12425 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.521561 12425 tablet_bootstrap.cc:492] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Bootstrap complete.
I20240503 13:59:45.521879 12425 ts_tablet_manager.cc:1367] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Time spent bootstrapping tablet: real 0.101s user 0.028s sys 0.000s
I20240503 13:59:45.522496 12425 raft_consensus.cc:348] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:45.522755 12425 raft_consensus.cc:729] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e5c0b4b872d451dba60797739f83576, State: Initialized, Role: LEARNER
I20240503 13:59:45.522914 12425 consensus_queue.cc:260] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }
I20240503 13:59:45.523326 12425 ts_tablet_manager.cc:1398] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.524363 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session 1e5c0b4b872d451dba60797739f83576-e0b7095bdf3549fb83b3573a8e65deee received from {username='slave'} at 127.0.99.130:58063
I20240503 13:59:45.524442 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session 1e5c0b4b872d451dba60797739f83576-e0b7095bdf3549fb83b3573a8e65deee on tablet e0b7095bdf3549fb83b3573a8e65deee with peer 1e5c0b4b872d451dba60797739f83576
W20240503 13:59:45.529626 7236 consensus_peers.cc:462] T 3622884b9b314ccca24422d8fa05360a P e1469ca6a5f945ec89ee1eb8df023b96 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111)
I20240503 13:59:45.551723 12660 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.555399 9469 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 13:59:37.856534 9476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.856809 9476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.860704 9476 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:38.905625 9476 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 13:59:39.366731 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.366883 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.366941 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.367019 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.367069 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.367111 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.367179 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.367252 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.367307 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.367369 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.367416 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.367448 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.367494 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.367537 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.367579 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.367616 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
I20240503 13:59:39.443706 9476 rebalancer_tool.cc:1074] tablet 8aff688eaad64d149bdc1c8ab12f4ca1: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:39.516238 9476 rebalancer_tool.cc:1074] tablet b23f9eeeb4af45b7b679b70514e247b8: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.566972 9476 rebalancer_tool.cc:1074] tablet 78bb6d55e8b44732b32da55f00071b21: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:39.611948 9476 rebalancer_tool.cc:1074] tablet 33b7b43742e34431af0c16d320b35dd7: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.647682 9476 rebalancer_tool.cc:1074] tablet e0b7095bdf3549fb83b3573a8e65deee: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:39.681821 9476 rebalancer_tool.cc:1074] tablet 902cf3db3581465c83170c4b13b3bffb: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.730572 9476 rebalancer_tool.cc:1074] tablet f73a5db0cae647d4b3375e6d42358307: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:39.798993 9476 rebalancer_tool.cc:1084] tablet c74f828c28c64097ab4bf5ec66459d2f: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet c74f828c28c64097ab4bf5ec66459d2f: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:39.799105 9476 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:40.307636 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.317836 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:40.317927 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.318853 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.318928 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:40.318982 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.319161 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.319453 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.319711 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.319881 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.320012 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.320930 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:40.496176 9476 rebalancer_tool.cc:1074] tablet c1a85f321bfa4faf9b48cb319221fb94: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:40.594115 9476 rebalancer_tool.cc:1084] tablet 4e2cabe02dbf4580b53875aaa09aaffa: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 4e2cabe02dbf4580b53875aaa09aaffa: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:40.594209 9476 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:42.117863 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.117959 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.118014 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.118077 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:42.118142 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.118188 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.118230 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:42.118271 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.118316 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:42.118408 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.118446 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.118469 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.118499 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.118527 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.118558 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.118659 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.118719 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.118762 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:42.240689 9476 rebalancer_tool.cc:1074] tablet 9b04204ad281452794d724fb4d7212f8: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:42.326896 9476 rebalancer_tool.cc:1084] tablet 23cf5703ccef47ea927342b9c29f820a: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 23cf5703ccef47ea927342b9c29f820a: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:42.326999 9476 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.076771 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.076902 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server e1469ca6a5f945ec89ee1eb8df023b96
W20240503 13:59:43.076957 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server da390a1a407a4f30b217b3d60fba563f to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:43.076999 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:43.077035 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.077073 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:43.184762 9476 rebalancer_tool.cc:1084] tablet 14304180134241c792ec1f23ff89f94c: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '' move ignored: Not found: Peer with uuid 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in consensus config
I20240503 13:59:43.184872 9476 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.771836 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:43.772044 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.772096 9476 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
I20240503 13:59:43.935145 9476 rebalancer_tool.cc:1074] tablet 3622884b9b314ccca24422d8fa05360a: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:43.979607 9476 rebalancer_tool.cc:1074] tablet 63552cb8c9c94a07b2a0b157402efa7a: '9d921974718b4613b136eee9d0e1bad2' -> '1e5c0b4b872d451dba60797739f83576' move scheduled
I20240503 13:59:44.219080 9476 rebalancer_tool.cc:1074] tablet 86634d81241c44cebcb602a6982f4244: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:44.326594 9476 rebalancer_tool.cc:1074] tablet 2212cf72b6ee4ee5b086455a75482e3b: '708734e6729441fcb2c5952fa47771d9' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:44.379671 9476 rebalancer_tool.cc:1074] tablet a4b0fd771f844fcab38aec069b9abd00: '1e5c0b4b872d451dba60797739f83576' -> '7a6f98c1ec0346ebb30c59ffe8d7c29a' move scheduled
I20240503 13:59:44.448539 9476 rebalancer_tool.cc:1074] tablet 9445bc10fdcf4e9b8d9167ba243bfda5: '1e5c0b4b872d451dba60797739f83576' -> 'e1469ca6a5f945ec89ee1eb8df023b96' move scheduled
I20240503 13:59:44.721268 9476 rebalancer_tool.cc:1136] tablet c1a85f321bfa4faf9b48cb319221fb94: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: OK
I20240503 13:59:44.756373 9476 rebalancer_tool.cc:1123] tablet b23f9eeeb4af45b7b679b70514e247b8: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move is abandoned: Illegal state: b23f9eeeb4af45b7b679b70514e247b8: source replica e1469ca6a5f945ec89ee1eb8df023b96 does not have REPLACE attribute set
I20240503 13:59:44.768602 9476 rebalancer_tool.cc:1123] tablet 78bb6d55e8b44732b32da55f00071b21: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move is abandoned: Illegal state: 78bb6d55e8b44732b32da55f00071b21: source replica 9d921974718b4613b136eee9d0e1bad2 does not have REPLACE attribute set
I20240503 13:59:44.783088 9476 rebalancer_tool.cc:1136] tablet 33b7b43742e34431af0c16d320b35dd7: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: OK
I20240503 13:59:44.804142 9476 rebalancer_tool.cc:1123] tablet e0b7095bdf3549fb83b3573a8e65deee: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move is abandoned: Illegal state: e0b7095bdf3549fb83b3573a8e65deee: source replica 1e5c0b4b872d451dba60797739f83576 does not have REPLACE attribute set
I20240503 13:59:44.813705 9476 rebalancer_tool.cc:1136] tablet f73a5db0cae647d4b3375e6d42358307: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move completed: OK
I20240503 13:59:44.825078 9476 rebalancer_tool.cc:1136] tablet 8aff688eaad64d149bdc1c8ab12f4ca1: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move completed: OK
I20240503 13:59:44.840415 9476 rebalancer_tool.cc:1136] tablet 902cf3db3581465c83170c4b13b3bffb: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move completed: OK
I20240503 13:59:44.870165 9476 rebalancer_tool.cc:1084] tablet 23cf5703ccef47ea927342b9c29f820a: '1e5c0b4b872d451dba60797739f83576' -> '7a6f98c1ec0346ebb30c59ffe8d7c29a' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } }
I20240503 13:59:44.870270 9476 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:45.290958 12561 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:59:45.068642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:36553 (local address 127.0.0.1:44522)
0503 13:59:45.069237 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0503 13:59:45.069250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 13:59:45.069333 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 13:59:45.290222 (+220889us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":92,"threads_started":1}
W20240503 13:59:45.295694 12671 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:59:45.095341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:43485 (local address 127.0.0.1:50400)
0503 13:59:45.106371 (+ 11030us) negotiation.cc:107] Waiting for socket to connect
0503 13:59:45.106389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 13:59:45.106490 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 13:59:45.295600 (+189110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":10927}
Illegal state: tablet server c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): unacceptable health status UNAVAILABLE
W20240503 13:59:45.557590 7498 consensus_peers.cc:462] T 86634d81241c44cebcb602a6982f4244 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
I20240503 13:59:45.559824 12660 tablet_bootstrap.cc:492] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap complete.
I20240503 13:59:45.560127 12660 ts_tablet_manager.cc:1367] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.092s user 0.031s sys 0.001s
I20240503 13:59:45.560724 12660 raft_consensus.cc:348] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:45.559423 12563 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.561365 12563 tablet_bootstrap.cc:492] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap complete.
I20240503 13:59:45.561602 12563 ts_tablet_manager.cc:1367] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.131s user 0.027s sys 0.003s
I20240503 13:59:45.562276 12563 raft_consensus.cc:348] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:45.562491 12563 raft_consensus.cc:729] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: LEARNER
I20240503 13:59:45.562667 12563 consensus_queue.cc:260] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:45.563796 12563 ts_tablet_manager.cc:1398] T a4b0fd771f844fcab38aec069b9abd00 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:45.564419 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 7a6f98c1ec0346ebb30c59ffe8d7c29a-a4b0fd771f844fcab38aec069b9abd00 received from {username='slave'} at 127.0.99.132:52805
I20240503 13:59:45.564507 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 7a6f98c1ec0346ebb30c59ffe8d7c29a-a4b0fd771f844fcab38aec069b9abd00 on tablet a4b0fd771f844fcab38aec069b9abd00 with peer 7a6f98c1ec0346ebb30c59ffe8d7c29a
I20240503 13:59:45.560988 12660 raft_consensus.cc:729] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: LEARNER
I20240503 13:59:45.565796 12660 consensus_queue.cc:260] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:45.567407 12552 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.567778 12660 ts_tablet_manager.cc:1398] T 3622884b9b314ccca24422d8fa05360a P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:59:45.569384 12649 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.569701 12649 tablet_bootstrap.cc:492] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Bootstrap complete.
I20240503 13:59:45.569875 12649 ts_tablet_manager.cc:1367] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Time spent bootstrapping tablet: real 0.147s user 0.025s sys 0.000s
I20240503 13:59:45.569253 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 7a6f98c1ec0346ebb30c59ffe8d7c29a-3622884b9b314ccca24422d8fa05360a received from {username='slave'} at 127.0.99.132:52805
I20240503 13:59:45.572973 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 7a6f98c1ec0346ebb30c59ffe8d7c29a-3622884b9b314ccca24422d8fa05360a on tablet 3622884b9b314ccca24422d8fa05360a with peer 7a6f98c1ec0346ebb30c59ffe8d7c29a
I20240503 13:59:45.574144 12464 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.574532 12464 tablet_bootstrap.cc:492] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Bootstrap complete.
I20240503 13:59:45.574559 12552 tablet_bootstrap.cc:492] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: Bootstrap complete.
I20240503 13:59:45.574769 12464 ts_tablet_manager.cc:1367] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Time spent bootstrapping tablet: real 0.174s user 0.031s sys 0.000s
I20240503 13:59:45.574769 12552 ts_tablet_manager.cc:1367] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent bootstrapping tablet: real 0.155s user 0.025s sys 0.004s
I20240503 13:59:45.575238 12464 raft_consensus.cc:348] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:45.575238 12552 raft_consensus.cc:348] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:45.575374 12552 raft_consensus.cc:729] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1469ca6a5f945ec89ee1eb8df023b96, State: Initialized, Role: LEARNER
I20240503 13:59:45.575512 12464 raft_consensus.cc:729] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 708734e6729441fcb2c5952fa47771d9, State: Initialized, Role: LEARNER
I20240503 13:59:45.575481 12552 consensus_queue.cc:260] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { promote: true } }
I20240503 13:59:45.575690 12464 consensus_queue.cc:260] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:45.575825 12552 ts_tablet_manager.cc:1398] T 9445bc10fdcf4e9b8d9167ba243bfda5 P e1469ca6a5f945ec89ee1eb8df023b96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:45.576143 12464 ts_tablet_manager.cc:1398] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.577733 7450 tablet_copy_service.cc:342] P 1e5c0b4b872d451dba60797739f83576: Request end of tablet copy session 708734e6729441fcb2c5952fa47771d9-fca185e6c04648c5aa4f4ad51b0e1a39 received from {username='slave'} at 127.0.99.134:47471
I20240503 13:59:45.577814 7450 tablet_copy_service.cc:433] P 1e5c0b4b872d451dba60797739f83576: ending tablet copy session 708734e6729441fcb2c5952fa47771d9-fca185e6c04648c5aa4f4ad51b0e1a39 on tablet fca185e6c04648c5aa4f4ad51b0e1a39 with peer 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:45.578461 12649 raft_consensus.cc:348] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } }
I20240503 13:59:45.578974 7710 tablet_copy_service.cc:342] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Request end of tablet copy session e1469ca6a5f945ec89ee1eb8df023b96-9445bc10fdcf4e9b8d9167ba243bfda5 received from {username='slave'} at 127.0.99.129:43167
I20240503 13:59:45.579048 7710 tablet_copy_service.cc:433] P 7a6f98c1ec0346ebb30c59ffe8d7c29a: ending tablet copy session e1469ca6a5f945ec89ee1eb8df023b96-9445bc10fdcf4e9b8d9167ba243bfda5 on tablet 9445bc10fdcf4e9b8d9167ba243bfda5 with peer e1469ca6a5f945ec89ee1eb8df023b96
I20240503 13:59:45.580070 12649 raft_consensus.cc:729] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Initialized, Role: LEARNER
I20240503 13:59:45.580286 12649 consensus_queue.cc:260] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { promote: true } }
I20240503 13:59:45.580683 12649 ts_tablet_manager.cc:1398] T 78bb6d55e8b44732b32da55f00071b21 P 9d921974718b4613b136eee9d0e1bad2: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240503 13:59:45.581729 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 9d921974718b4613b136eee9d0e1bad2-78bb6d55e8b44732b32da55f00071b21 received from {username='slave'} at 127.0.99.131:48917
I20240503 13:59:45.581818 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 9d921974718b4613b136eee9d0e1bad2-78bb6d55e8b44732b32da55f00071b21 on tablet 78bb6d55e8b44732b32da55f00071b21 with peer 9d921974718b4613b136eee9d0e1bad2
I20240503 13:59:45.582537 12553 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:45.582921 12553 tablet_bootstrap.cc:492] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Bootstrap complete.
I20240503 13:59:45.583211 12553 ts_tablet_manager.cc:1367] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent bootstrapping tablet: real 0.139s user 0.032s sys 0.000s
I20240503 13:59:45.583900 12553 raft_consensus.cc:348] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:45.584108 12553 raft_consensus.cc:729] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a6f98c1ec0346ebb30c59ffe8d7c29a, State: Initialized, Role: LEARNER
I20240503 13:59:45.584285 12553 consensus_queue.cc:260] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } attrs { replace: true } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { promote: true } }
I20240503 13:59:45.586198 12553 ts_tablet_manager.cc:1398] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:45.588188 7320 tablet_copy_service.cc:342] P e1469ca6a5f945ec89ee1eb8df023b96: Request end of tablet copy session 7a6f98c1ec0346ebb30c59ffe8d7c29a-f3eaa7e5f641436a8f0cfdaf83443ca1 received from {username='slave'} at 127.0.99.132:52805
I20240503 13:59:45.588270 7320 tablet_copy_service.cc:433] P e1469ca6a5f945ec89ee1eb8df023b96: ending tablet copy session 7a6f98c1ec0346ebb30c59ffe8d7c29a-f3eaa7e5f641436a8f0cfdaf83443ca1 on tablet f3eaa7e5f641436a8f0cfdaf83443ca1 with peer 7a6f98c1ec0346ebb30c59ffe8d7c29a
I20240503 13:59:45.595108 7423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "902cf3db3581465c83170c4b13b3bffb"
dest_uuid: "1e5c0b4b872d451dba60797739f83576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41744
I20240503 13:59:45.595290 7423 raft_consensus.cc:595] T 902cf3db3581465c83170c4b13b3bffb P 1e5c0b4b872d451dba60797739f83576 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:45.600908 9470 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 13:59:37.848758 9475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.849059 9475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.854027 9475 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:39.004257 9475 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 13:59:39.689538 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.690163 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.690328 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.690393 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.690480 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.690584 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.690624 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.690658 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.690694 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.690797 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.690887 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.690948 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.690995 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.691041 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.691087 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.691150 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.691221 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.691285 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.691349 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.691406 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.691449 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:39.745144 9475 rebalancer_tool.cc:1074] tablet 14304180134241c792ec1f23ff89f94c: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.805392 9475 rebalancer_tool.cc:1074] tablet fca185e6c04648c5aa4f4ad51b0e1a39: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:39.855618 9475 rebalancer_tool.cc:1074] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:39.891566 9475 rebalancer_tool.cc:1074] tablet c74f828c28c64097ab4bf5ec66459d2f: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.912590 9475 rebalancer_tool.cc:1074] tablet 902cf3db3581465c83170c4b13b3bffb: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.994302 9475 rebalancer_tool.cc:1074] tablet 2212cf72b6ee4ee5b086455a75482e3b: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:40.144183 9475 rebalancer_tool.cc:1074] tablet 33b7b43742e34431af0c16d320b35dd7: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:40.249902 9475 rebalancer_tool.cc:1074] tablet a4b0fd771f844fcab38aec069b9abd00: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:40.267886 9475 rebalancer_tool.cc:1074] tablet 4e2cabe02dbf4580b53875aaa09aaffa: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:40.452832 9475 rebalancer_tool.cc:1074] tablet 4dffd8718851475099a6696e4a2b8901: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:40.539992 9475 rebalancer_tool.cc:1074] tablet faa9fb136138439b8f694ac95dbfa2e7: '1e5c0b4b872d451dba60797739f83576' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:40.658609 9475 rebalancer_tool.cc:1084] tablet 86634d81241c44cebcb602a6982f4244: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 86634d81241c44cebcb602a6982f4244: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:40.658720 9475 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:41.990558 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.990645 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.990674 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:41.990736 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.990761 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.990793 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.990831 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.001559 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.001677 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.001726 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.001767 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:42.001808 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.001853 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.001894 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.001933 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:42.238359 9475 rebalancer_tool.cc:1074] tablet 93d454886dfa40018a4403a3d74b3de2: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:42.322382 9475 rebalancer_tool.cc:1084] tablet 23cf5703ccef47ea927342b9c29f820a: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 23cf5703ccef47ea927342b9c29f820a: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:42.322481 9475 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.250798 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.250876 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:43.250909 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server da390a1a407a4f30b217b3d60fba563f to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.251017 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server c5c3042421284f028fa512b7876da405 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:43.251093 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.251140 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:43.251262 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
I20240503 13:59:43.304334 9475 rebalancer_tool.cc:1074] tablet 23cf5703ccef47ea927342b9c29f820a: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '1e5c0b4b872d451dba60797739f83576' move scheduled
I20240503 13:59:43.359642 9475 rebalancer_tool.cc:1074] tablet f73a5db0cae647d4b3375e6d42358307: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:43.467402 9475 rebalancer_tool.cc:1074] tablet ade40ddaf0304b1a9e94db6630175c0e: '708734e6729441fcb2c5952fa47771d9' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:43.615747 9475 rebalancer_tool.cc:1084] tablet b23f9eeeb4af45b7b679b70514e247b8: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'e1469ca6a5f945ec89ee1eb8df023b96' move ignored: Illegal state: tablet b23f9eeeb4af45b7b679b70514e247b8: replica e1469ca6a5f945ec89ee1eb8df023b96 is already present
I20240503 13:59:43.615835 9475 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:44.185446 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:44.185580 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:44.185634 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:44.185693 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 476f53c7e40c4a0b86ffbd818aea818d to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:44.185758 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server c5c3042421284f028fa512b7876da405 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:44.185807 9475 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server e1469ca6a5f945ec89ee1eb8df023b96
I20240503 13:59:44.277885 9475 rebalancer_tool.cc:1074] tablet e0b7095bdf3549fb83b3573a8e65deee: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '1e5c0b4b872d451dba60797739f83576' move scheduled
I20240503 13:59:44.405229 9475 rebalancer_tool.cc:1136] tablet 93d454886dfa40018a4403a3d74b3de2: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move completed: OK
I20240503 13:59:44.462092 9475 rebalancer_tool.cc:1136] tablet faa9fb136138439b8f694ac95dbfa2e7: '1e5c0b4b872d451dba60797739f83576' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: Incomplete: tablet faa9fb136138439b8f694ac95dbfa2e7, TS 1e5c0b4b872d451dba60797739f83576 -> TS da390a1a407a4f30b217b3d60fba563f move failed, target replica disappeared
I20240503 13:59:44.475337 9475 rebalancer_tool.cc:1136] tablet 4dffd8718851475099a6696e4a2b8901: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move completed: OK
I20240503 13:59:44.518389 9475 rebalancer_tool.cc:1136] tablet 33b7b43742e34431af0c16d320b35dd7: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move completed: Incomplete: tablet 33b7b43742e34431af0c16d320b35dd7, TS 1e5c0b4b872d451dba60797739f83576 -> TS 476f53c7e40c4a0b86ffbd818aea818d move failed, target replica disappeared
I20240503 13:59:44.554819 9475 rebalancer_tool.cc:1123] tablet a4b0fd771f844fcab38aec069b9abd00: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move is abandoned: Illegal state: a4b0fd771f844fcab38aec069b9abd00: source replica 7a6f98c1ec0346ebb30c59ffe8d7c29a does not have REPLACE attribute set
I20240503 13:59:44.834957 9475 rebalancer_tool.cc:1136] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move completed: OK
I20240503 13:59:44.858180 9475 rebalancer_tool.cc:1136] tablet c74f828c28c64097ab4bf5ec66459d2f: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'c5c3042421284f028fa512b7876da405' move completed: OK
I20240503 13:59:44.895453 9475 rebalancer_tool.cc:1136] tablet 902cf3db3581465c83170c4b13b3bffb: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: OK
I20240503 13:59:44.929718 9475 rebalancer_tool.cc:1136] tablet 2212cf72b6ee4ee5b086455a75482e3b: '9d921974718b4613b136eee9d0e1bad2' -> '708734e6729441fcb2c5952fa47771d9' move completed: OK
I20240503 13:59:44.952014 9475 rebalancer_tool.cc:1084] tablet 78bb6d55e8b44732b32da55f00071b21: '1e5c0b4b872d451dba60797739f83576' -> '9d921974718b4613b136eee9d0e1bad2' move ignored: Illegal state: tablet 78bb6d55e8b44732b32da55f00071b21: replica 9d921974718b4613b136eee9d0e1bad2 is already present
I20240503 13:59:44.952103 9475 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:45.292132 12734 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:59:45.204459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:36553 (local address 127.0.0.1:44530)
0503 13:59:45.238931 (+ 34472us) negotiation.cc:107] Waiting for socket to connect
0503 13:59:45.238946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 13:59:45.239025 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 13:59:45.291339 (+ 52314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":34399}
W20240503 13:59:45.296617 12732 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:59:45.203750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:43485 (local address 127.0.0.1:50412)
0503 13:59:45.211255 (+ 7505us) negotiation.cc:107] Waiting for socket to connect
0503 13:59:45.211269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 13:59:45.211357 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 13:59:45.296484 (+ 85127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7448}
Illegal state: tablet server c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): unacceptable health status UNAVAILABLE
I20240503 13:59:45.618904 7561 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d"
dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41102
I20240503 13:59:45.619098 7561 raft_consensus.cc:595] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:45.629416 7498 consensus_peers.cc:462] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
I20240503 13:59:45.632189 7690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23f9eeeb4af45b7b679b70514e247b8"
dest_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33506
I20240503 13:59:45.632334 7690 raft_consensus.cc:595] T b23f9eeeb4af45b7b679b70514e247b8 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:45.641108 7561 consensus_queue.cc:237] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } }
I20240503 13:59:45.643198 9154 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 13:59:45.643304 7690 raft_consensus.cc:1261] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 13:59:45.644001 7498 consensus_peers.cc:485] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Couldn't send request to peer 476f53c7e40c4a0b86ffbd818aea818d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.644045 12721 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:45.644371 12719 consensus_queue.cc:1035] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 13:59:45.644836 7498 consensus_peers.cc:485] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.646309 10201 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:45.646677 7692 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:45.649247 7153 catalog_manager.cc:5484] T faa9fb136138439b8f694ac95dbfa2e7 P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 10 to 11, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:45.649302 9154 raft_consensus.cc:2932] T faa9fb136138439b8f694ac95dbfa2e7 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: true } } }
I20240503 13:59:45.664140 12721 raft_consensus.cc:979] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2: : Instructing follower e1469ca6a5f945ec89ee1eb8df023b96 to start an election
I20240503 13:59:45.664166 12720 raft_consensus.cc:1050] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 1e5c0b4b872d451dba60797739f83576 to VOTER
I20240503 13:59:45.664274 12721 raft_consensus.cc:1067] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Signalling peer e1469ca6a5f945ec89ee1eb8df023b96 to start an election
I20240503 13:59:45.664671 12720 consensus_queue.cc:237] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } }
I20240503 13:59:45.665109 7297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d"
dest_uuid: "e1469ca6a5f945ec89ee1eb8df023b96"
from {username='slave'} at 127.0.99.131:48917
I20240503 13:59:45.665278 7297 raft_consensus.cc:482] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:45.665344 7297 raft_consensus.cc:3037] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.666357 9154 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:45.666592 7297 raft_consensus.cc:504] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:45.666800 7692 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:45.666874 7423 raft_consensus.cc:1261] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:45.667155 7297 leader_election.cc:288] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 2 election: Requested vote from peers 9d921974718b4613b136eee9d0e1bad2 (127.0.99.131:44401), 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553), da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485)
I20240503 13:59:45.667234 12720 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.667596 12720 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.668639 10201 consensus_queue.cc:1035] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.668943 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110ac05b2e8c4594a03225ab2c1ad56d" candidate_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "9d921974718b4613b136eee9d0e1bad2"
I20240503 13:59:45.669056 7561 raft_consensus.cc:3032] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:45.669097 7561 raft_consensus.cc:729] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d921974718b4613b136eee9d0e1bad2, State: Running, Role: LEADER
I20240503 13:59:45.669260 7561 consensus_queue.cc:260] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "476f53c7e40c4a0b86ffbd818aea818d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36553 } attrs { promote: false } } peers { permanent_uuid: "da390a1a407a4f30b217b3d60fba563f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43485 } attrs { promote: false } }
I20240503 13:59:45.669469 7561 raft_consensus.cc:3037] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:45.670673 7561 raft_consensus.cc:2454] T 110ac05b2e8c4594a03225ab2c1ad56d P 9d921974718b4613b136eee9d0e1bad2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1469ca6a5f945ec89ee1eb8df023b96 in term 2.
W20240503 13:59:45.672436 7236 leader_election.cc:334] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111)
W20240503 13:59:45.672586 7236 leader_election.cc:334] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
I20240503 13:59:45.672665 7236 leader_election.cc:302] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 9d921974718b4613b136eee9d0e1bad2, e1469ca6a5f945ec89ee1eb8df023b96; no voters: 476f53c7e40c4a0b86ffbd818aea818d, da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:45.672855 10276 raft_consensus.cc:2726] T 110ac05b2e8c4594a03225ab2c1ad56d P e1469ca6a5f945ec89ee1eb8df023b96 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 13:59:45.674104 12721 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:45.674593 9154 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:45.674803 7423 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:45.676319 7692 raft_consensus.cc:2932] T 63552cb8c9c94a07b2a0b157402efa7a P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } } }
I20240503 13:59:45.677544 7149 catalog_manager.cc:5484] T 63552cb8c9c94a07b2a0b157402efa7a P 9d921974718b4613b136eee9d0e1bad2 reported cstate change: config changed from index 7 to 8, 1e5c0b4b872d451dba60797739f83576 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:45.690558 7367 consensus_peers.cc:462] T 4dffd8718851475099a6696e4a2b8901 P 1e5c0b4b872d451dba60797739f83576 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111)
I20240503 13:59:45.702409 7423 raft_consensus.cc:1203] T e0b7095bdf3549fb83b3573a8e65deee P 1e5c0b4b872d451dba60797739f83576 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 13:59:45.722399 7673 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:45.724440 9136 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:45.735577 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:45.746490 7278 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 13:59:45.761745 7498 consensus_peers.cc:462] T a1115afa5ead4c95aea379ad9987cbe0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:36553: connect: Connection refused (error 111)
I20240503 13:59:45.767957 7542 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:45.768043 10197 raft_consensus.cc:1050] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2: attempting to promote NON_VOTER 708734e6729441fcb2c5952fa47771d9 to VOTER
I20240503 13:59:45.768488 10197 consensus_queue.cc:237] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } }
I20240503 13:59:45.770783 9154 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:45.771221 7423 raft_consensus.cc:1261] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Refusing update from remote peer 9d921974718b4613b136eee9d0e1bad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:45.771809 12718 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:45.772135 12721 consensus_queue.cc:1035] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 13:59:45.773885 7498 consensus_peers.cc:485] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 -> Peer c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): Couldn't send request to peer c5c3042421284f028fa512b7876da405. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.774569 12721 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 9d921974718b4613b136eee9d0e1bad2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:45.775022 9154 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:45.777622 7153 catalog_manager.cc:5484] T 491ee8399ef04ae7b93e3228fef05eb0 P 708734e6729441fcb2c5952fa47771d9 reported cstate change: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9d921974718b4613b136eee9d0e1bad2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:45.777801 7423 raft_consensus.cc:2932] T 491ee8399ef04ae7b93e3228fef05eb0 P 1e5c0b4b872d451dba60797739f83576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 708734e6729441fcb2c5952fa47771d9 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: false } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: false } } }
I20240503 13:59:45.795151 7692 raft_consensus.cc:1203] T f3eaa7e5f641436a8f0cfdaf83443ca1 P 7a6f98c1ec0346ebb30c59ffe8d7c29a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 13:59:45.796247 9136 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:45.798817 7542 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:45.803956 7278 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 13:59:45.811461 9092 consensus_peers.cc:485] T ff246d9b2c594282bf2e99d2fd2b617d P 708734e6729441fcb2c5952fa47771d9 -> Peer da390a1a407a4f30b217b3d60fba563f (127.0.99.136:43485): Couldn't send request to peer da390a1a407a4f30b217b3d60fba563f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.823055 7673 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:45.830652 7410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:45.835960 9472 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 13:59:37.804601 9473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.804875 9473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.809309 9473 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:38.982213 9473 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 13:59:39.471410 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.471572 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.471637 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.471709 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.471818 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.471892 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.471969 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.472028 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.472069 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.472105 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.472787 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.472863 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.472911 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.472960 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:39.559190 9473 rebalancer_tool.cc:1074] tablet e0b7095bdf3549fb83b3573a8e65deee: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.589725 9473 rebalancer_tool.cc:1074] tablet b23f9eeeb4af45b7b679b70514e247b8: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.609212 9473 rebalancer_tool.cc:1084] tablet 8aff688eaad64d149bdc1c8ab12f4ca1: '1e5c0b4b872d451dba60797739f83576' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 8aff688eaad64d149bdc1c8ab12f4ca1: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:39.609310 9473 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:40.230726 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.231051 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.259694 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.259832 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.259894 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.259944 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:40.260006 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.260053 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.260100 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:40.260151 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:40.260205 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:40.260251 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:40.260301 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:40.293376 9473 rebalancer_tool.cc:1074] tablet 86634d81241c44cebcb602a6982f4244: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:40.332552 9473 rebalancer_tool.cc:1074] tablet faa9fb136138439b8f694ac95dbfa2e7: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:40.409528 9473 rebalancer_tool.cc:1074] tablet 4e2cabe02dbf4580b53875aaa09aaffa: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:40.504601 9473 rebalancer_tool.cc:1074] tablet 2212cf72b6ee4ee5b086455a75482e3b: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:40.573613 9473 rebalancer_tool.cc:1074] tablet 3622884b9b314ccca24422d8fa05360a: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:40.662034 9473 rebalancer_tool.cc:1084] tablet 14304180134241c792ec1f23ff89f94c: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move ignored: Illegal state: tablet 14304180134241c792ec1f23ff89f94c: replica 708734e6729441fcb2c5952fa47771d9 is already present
I20240503 13:59:40.662123 9473 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:42.020714 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.024456 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.025456 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.025563 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.025789 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.038228 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:42.038328 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.038380 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.038426 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:42.038470 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:42.038515 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.038560 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.038637 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:42.225327 9473 rebalancer_tool.cc:1084] tablet 9b04204ad281452794d724fb4d7212f8: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '708734e6729441fcb2c5952fa47771d9' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "1e5c0b4b872d451dba60797739f83576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46237 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "708734e6729441fcb2c5952fa47771d9" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36383 } attrs { promote: true } }
I20240503 13:59:42.225409 9473 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:42.969034 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:42.970726 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:42.970850 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server e1469ca6a5f945ec89ee1eb8df023b96
W20240503 13:59:42.970964 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:43.060863 9473 rebalancer_tool.cc:1074] tablet 33b7b43742e34431af0c16d320b35dd7: '1e5c0b4b872d451dba60797739f83576' -> '' move scheduled
I20240503 13:59:43.107555 9473 rebalancer_tool.cc:1074] tablet ff246d9b2c594282bf2e99d2fd2b617d: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:43.186435 9473 rebalancer_tool.cc:1084] tablet a1115afa5ead4c95aea379ad9987cbe0: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } attrs { replace: true } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:43.186537 9473 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.667939 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server e1469ca6a5f945ec89ee1eb8df023b96
W20240503 13:59:43.668115 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:43.668977 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.669101 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:43.669173 9473 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server c5c3042421284f028fa512b7876da405
I20240503 13:59:43.802492 9473 rebalancer_tool.cc:1074] tablet a4b0fd771f844fcab38aec069b9abd00: '1e5c0b4b872d451dba60797739f83576' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:43.892884 9473 rebalancer_tool.cc:1074] tablet 9b04204ad281452794d724fb4d7212f8: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:44.028446 9473 rebalancer_tool.cc:1074] tablet a1115afa5ead4c95aea379ad9987cbe0: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:44.158774 9473 rebalancer_tool.cc:1074] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '9d921974718b4613b136eee9d0e1bad2' -> '1e5c0b4b872d451dba60797739f83576' move scheduled
I20240503 13:59:44.221071 9473 rebalancer_tool.cc:1074] tablet 491ee8399ef04ae7b93e3228fef05eb0: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:44.303287 9473 rebalancer_tool.cc:1074] tablet 4dffd8718851475099a6696e4a2b8901: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:44.420701 9473 rebalancer_tool.cc:1074] tablet f3eaa7e5f641436a8f0cfdaf83443ca1: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '7a6f98c1ec0346ebb30c59ffe8d7c29a' move scheduled
I20240503 13:59:44.453243 9473 rebalancer_tool.cc:1136] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '9d921974718b4613b136eee9d0e1bad2' -> '1e5c0b4b872d451dba60797739f83576' move completed: Incomplete: tablet 110ac05b2e8c4594a03225ab2c1ad56d, TS 9d921974718b4613b136eee9d0e1bad2 -> TS 1e5c0b4b872d451dba60797739f83576 move failed, target replica disappeared
I20240503 13:59:44.610282 9473 rebalancer_tool.cc:1123] tablet 3622884b9b314ccca24422d8fa05360a: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'da390a1a407a4f30b217b3d60fba563f' move is abandoned: Illegal state: 3622884b9b314ccca24422d8fa05360a: source replica 7a6f98c1ec0346ebb30c59ffe8d7c29a does not have REPLACE attribute set
I20240503 13:59:44.675565 9473 rebalancer_tool.cc:1136] tablet b23f9eeeb4af45b7b679b70514e247b8: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: Incomplete: tablet b23f9eeeb4af45b7b679b70514e247b8, TS e1469ca6a5f945ec89ee1eb8df023b96 -> TS da390a1a407a4f30b217b3d60fba563f move failed, target replica disappeared
I20240503 13:59:45.561297 9473 rebalancer_tool.cc:1136] tablet 86634d81241c44cebcb602a6982f4244: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move completed: OK
I20240503 13:59:45.593235 9473 rebalancer_tool.cc:1136] tablet e0b7095bdf3549fb83b3573a8e65deee: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move completed: OK
I20240503 13:59:45.602216 9473 rebalancer_tool.cc:1136] tablet faa9fb136138439b8f694ac95dbfa2e7: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move completed: OK
I20240503 13:59:45.608541 9473 rebalancer_tool.cc:1136] tablet 4e2cabe02dbf4580b53875aaa09aaffa: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move completed: OK
I20240503 13:59:45.622632 9473 rebalancer_tool.cc:1084] tablet 9445bc10fdcf4e9b8d9167ba243bfda5: '1e5c0b4b872d451dba60797739f83576' -> 'e1469ca6a5f945ec89ee1eb8df023b96' move ignored: Illegal state: tablet 9445bc10fdcf4e9b8d9167ba243bfda5: replica e1469ca6a5f945ec89ee1eb8df023b96 is already present
I20240503 13:59:45.622723 9473 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server c5c3042421284f028fa512b7876da405 (127.0.99.135:41051): unacceptable health status UNAVAILABLE
I20240503 13:59:45.850910 9154 raft_consensus.cc:1203] T fca185e6c04648c5aa4f4ad51b0e1a39 P 708734e6729441fcb2c5952fa47771d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 13:59:45.867293 9468 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 13:59:37.841770 9477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.842211 9477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.846519 9477 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:38.986176 9477 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 13:59:39.463171 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.463440 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.463523 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.463579 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:39.463675 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.463742 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.463795 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.463842 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.463941 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.463996 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.464057 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:39.464141 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:39.464200 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:39.464272 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
I20240503 13:59:39.501777 9477 rebalancer_tool.cc:1074] tablet 63552cb8c9c94a07b2a0b157402efa7a: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:39.667189 9477 rebalancer_tool.cc:1074] tablet 4582c127b7d34e08a68ea9094de11f19: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:39.708245 9477 rebalancer_tool.cc:1074] tablet 2212cf72b6ee4ee5b086455a75482e3b: '9d921974718b4613b136eee9d0e1bad2' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.739466 9477 rebalancer_tool.cc:1074] tablet f3eaa7e5f641436a8f0cfdaf83443ca1: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:39.797191 9477 rebalancer_tool.cc:1074] tablet ade40ddaf0304b1a9e94db6630175c0e: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:39.836508 9477 rebalancer_tool.cc:1074] tablet ff246d9b2c594282bf2e99d2fd2b617d: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:39.983276 9477 rebalancer_tool.cc:1074] tablet fca185e6c04648c5aa4f4ad51b0e1a39: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:40.312613 9477 rebalancer_tool.cc:1074] tablet c1a85f321bfa4faf9b48cb319221fb94: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:40.488816 9477 rebalancer_tool.cc:1074] tablet 14304180134241c792ec1f23ff89f94c: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:40.544675 9477 rebalancer_tool.cc:1084] tablet 4e2cabe02dbf4580b53875aaa09aaffa: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move ignored: Illegal state: tablet 4e2cabe02dbf4580b53875aaa09aaffa: replica 476f53c7e40c4a0b86ffbd818aea818d is already present
I20240503 13:59:40.544770 9477 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:41.819501 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.822563 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.822667 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.822710 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.822791 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.822842 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.822881 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.822940 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.822990 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.823032 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.823069 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.823119 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
W20240503 13:59:41.824442 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:41.824692 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:41.824766 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
I20240503 13:59:41.955829 9477 rebalancer_tool.cc:1074] tablet 110ac05b2e8c4594a03225ab2c1ad56d: '9d921974718b4613b136eee9d0e1bad2' -> 'da390a1a407a4f30b217b3d60fba563f' move scheduled
I20240503 13:59:42.077775 9477 rebalancer_tool.cc:1074] tablet 23cf5703ccef47ea927342b9c29f820a: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:42.243090 9477 rebalancer_tool.cc:1084] tablet 93d454886dfa40018a4403a3d74b3de2: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9d921974718b4613b136eee9d0e1bad2" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44401 } attrs { replace: true } } peers { permanent_uuid: "7a6f98c1ec0346ebb30c59ffe8d7c29a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43133 } } peers { permanent_uuid: "e1469ca6a5f945ec89ee1eb8df023b96" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41881 } } peers { permanent_uuid: "c5c3042421284f028fa512b7876da405" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41051 } attrs { promote: true } }
I20240503 13:59:42.243196 9477 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:43.020452 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6bab62a3656442a6a25a416f35f12e63: could not find any suitable replica to move from server 708734e6729441fcb2c5952fa47771d9 to server c5c3042421284f028fa512b7876da405
W20240503 13:59:43.020536 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 581ba72cb36f49fabf7a67fb8d8d47ef: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:43.020573 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f48fee88cf048bdbeea42628a36c726: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:43.020602 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4211301acdd465c9d7742a40279a30c: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server e1469ca6a5f945ec89ee1eb8df023b96
W20240503 13:59:43.020774 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server e1469ca6a5f945ec89ee1eb8df023b96 to server da390a1a407a4f30b217b3d60fba563f
W20240503 13:59:43.020821 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 7a6f98c1ec0346ebb30c59ffe8d7c29a to server 476f53c7e40c4a0b86ffbd818aea818d
I20240503 13:59:43.048794 9477 rebalancer_tool.cc:1074] tablet 33b7b43742e34431af0c16d320b35dd7: '476f53c7e40c4a0b86ffbd818aea818d' -> '' move scheduled
I20240503 13:59:43.206955 9477 rebalancer_tool.cc:1074] tablet a1115afa5ead4c95aea379ad9987cbe0: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:43.258415 9477 rebalancer_tool.cc:1074] tablet 902cf3db3581465c83170c4b13b3bffb: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move scheduled
I20240503 13:59:43.305784 9477 rebalancer_tool.cc:1074] tablet b23f9eeeb4af45b7b679b70514e247b8: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> 'e1469ca6a5f945ec89ee1eb8df023b96' move scheduled
I20240503 13:59:43.393115 9477 rebalancer_tool.cc:1074] tablet 4e2cabe02dbf4580b53875aaa09aaffa: '1e5c0b4b872d451dba60797739f83576' -> 'c5c3042421284f028fa512b7876da405' move scheduled
I20240503 13:59:43.650756 9477 rebalancer_tool.cc:1074] tablet 93d454886dfa40018a4403a3d74b3de2: 'e1469ca6a5f945ec89ee1eb8df023b96' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:43.720278 9477 rebalancer_tool.cc:1084] tablet 3622884b9b314ccca24422d8fa05360a: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '708734e6729441fcb2c5952fa47771d9' move ignored: Not found: Peer with uuid 7a6f98c1ec0346ebb30c59ffe8d7c29a not found in consensus config
I20240503 13:59:43.720368 9477 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 13:59:44.495901 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a09ea72b63549d08f82a8a8cd0841b8: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:44.496011 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b8b322e08e5d48f39595d2b2ee79d218: could not find any suitable replica to move from server c5c3042421284f028fa512b7876da405 to server 7a6f98c1ec0346ebb30c59ffe8d7c29a
W20240503 13:59:44.496052 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5274a33c27e4111801d2bb4649f3d1a: could not find any suitable replica to move from server 476f53c7e40c4a0b86ffbd818aea818d to server 9d921974718b4613b136eee9d0e1bad2
W20240503 13:59:44.496088 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 371f6713b69b4d2baaff6174ec18e696: could not find any suitable replica to move from server 476f53c7e40c4a0b86ffbd818aea818d to server 1e5c0b4b872d451dba60797739f83576
W20240503 13:59:44.496160 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5b979a355c5c46acac1c80a82e9b5d33: could not find any suitable replica to move from server 9d921974718b4613b136eee9d0e1bad2 to server 708734e6729441fcb2c5952fa47771d9
W20240503 13:59:44.496201 9477 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3986eba04fca4c2fbdbe763bfcab3d02: could not find any suitable replica to move from server 1e5c0b4b872d451dba60797739f83576 to server 9d921974718b4613b136eee9d0e1bad2
I20240503 13:59:44.535051 9477 rebalancer_tool.cc:1074] tablet 3622884b9b314ccca24422d8fa05360a: 'c5c3042421284f028fa512b7876da405' -> '7a6f98c1ec0346ebb30c59ffe8d7c29a' move scheduled
I20240503 13:59:44.589601 9477 rebalancer_tool.cc:1136] tablet 14304180134241c792ec1f23ff89f94c: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move completed: OK
I20240503 13:59:44.704242 9477 rebalancer_tool.cc:1123] tablet f3eaa7e5f641436a8f0cfdaf83443ca1: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move is abandoned: Illegal state: f3eaa7e5f641436a8f0cfdaf83443ca1: source replica 7a6f98c1ec0346ebb30c59ffe8d7c29a does not have REPLACE attribute set
I20240503 13:59:44.798161 9477 rebalancer_tool.cc:1136] tablet ade40ddaf0304b1a9e94db6630175c0e: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move completed: Incomplete: tablet ade40ddaf0304b1a9e94db6630175c0e, TS 1e5c0b4b872d451dba60797739f83576 -> TS 708734e6729441fcb2c5952fa47771d9 move failed, target replica disappeared
I20240503 13:59:44.829063 9477 rebalancer_tool.cc:1136] tablet 2212cf72b6ee4ee5b086455a75482e3b: '9d921974718b4613b136eee9d0e1bad2' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: OK
I20240503 13:59:44.844126 9477 rebalancer_tool.cc:1136] tablet 4582c127b7d34e08a68ea9094de11f19: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'c5c3042421284f028fa512b7876da405' move completed: OK
I20240503 13:59:44.905416 9477 rebalancer_tool.cc:1136] tablet ff246d9b2c594282bf2e99d2fd2b617d: 'e1469ca6a5f945ec89ee1eb8df023b96' -> 'da390a1a407a4f30b217b3d60fba563f' move completed: OK
I20240503 13:59:44.940214 9477 rebalancer_tool.cc:1123] tablet 63552cb8c9c94a07b2a0b157402efa7a: '1e5c0b4b872d451dba60797739f83576' -> '708734e6729441fcb2c5952fa47771d9' move is abandoned: Illegal state: 63552cb8c9c94a07b2a0b157402efa7a: source replica 1e5c0b4b872d451dba60797739f83576 does not have REPLACE attribute set
I20240503 13:59:45.253218 9477 rebalancer_tool.cc:1136] tablet fca185e6c04648c5aa4f4ad51b0e1a39: '9d921974718b4613b136eee9d0e1bad2' -> 'c5c3042421284f028fa512b7876da405' move completed: OK
I20240503 13:59:45.450843 9477 rebalancer_tool.cc:1136] tablet c1a85f321bfa4faf9b48cb319221fb94: '7a6f98c1ec0346ebb30c59ffe8d7c29a' -> '476f53c7e40c4a0b86ffbd818aea818d' move completed: OK
I20240503 13:59:45.606926 9477 rebalancer_tool.cc:1136] tablet 33b7b43742e34431af0c16d320b35dd7: '476f53c7e40c4a0b86ffbd818aea818d' -> '' move completed: OK
I20240503 13:59:45.650110 9477 rebalancer_tool.cc:1074] tablet faa9fb136138439b8f694ac95dbfa2e7: '708734e6729441fcb2c5952fa47771d9' -> '476f53c7e40c4a0b86ffbd818aea818d' move scheduled
I20240503 13:59:45.658778 9477 rebalancer_tool.cc:1084] tablet 9445bc10fdcf4e9b8d9167ba243bfda5: '1e5c0b4b872d451dba60797739f83576' -> 'e1469ca6a5f945ec89ee1eb8df023b96' move ignored: Illegal state: tablet 9445bc10fdcf4e9b8d9167ba243bfda5: replica e1469ca6a5f945ec89ee1eb8df023b96 is already present
I20240503 13:59:45.658885 9477 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 476f53c7e40c4a0b86ffbd818aea818d (127.0.99.133:36553): unacceptable health status UNAVAILABLE
E20240503 13:59:45.867700 398 external_mini_cluster.cc:943] Process with UUID 476f53c7e40c4a0b86ffbd818aea818d has crashed
E20240503 13:59:45.867830 398 external_mini_cluster.cc:943] Process with UUID c5c3042421284f028fa512b7876da405 has crashed
E20240503 13:59:45.867905 398 external_mini_cluster.cc:943] Process with UUID da390a1a407a4f30b217b3d60fba563f has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 3
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 13:59:45.868629 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7218
I20240503 13:59:45.876854 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7349
I20240503 13:59:45.884225 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7480
I20240503 13:59:45.891489 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7611
W20240503 13:59:45.894325 9090 consensus_peers.cc:485] T 9b04204ad281452794d724fb4d7212f8 P 708734e6729441fcb2c5952fa47771d9 -> Peer 1e5c0b4b872d451dba60797739f83576 (127.0.99.130:46237): Couldn't send request to peer 1e5c0b4b872d451dba60797739f83576. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:45.898583 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 9074
I20240503 13:59:45.905203 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 7127
2024-05-03T13:59:45Z chronyd exiting
I20240503 13:59:45.922514 398 test_util.cc:175] -----------------------------------------------
I20240503 13:59:45.922593 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714744724484359-398-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (19987 ms)
[----------] 1 test from ConcurrentRebalancersTest (19987 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240503 13:59:45.923324 398 test_util.cc:268] Using random seed: 567831801
I20240503 13:59:45.924129 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:45.924173 398 ts_itest-base.cc:113] --------------
I20240503 13:59:45.924185 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 13:59:45.924196 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 13:59:45.924206 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:45Z Disabled control of system clock
I20240503 13:59:45.932785 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42483
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42483
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 13:59:46.043995 12970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:46.044286 12970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:46.044344 12970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:46.047513 12970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:46.047600 12970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:46.047633 12970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:46.047657 12970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:46.047693 12970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:46.051622 12970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42483
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:46.052659 12970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:46.054086 12970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:46.062666 12970 server_base.cc:1034] running on GCE node
W20240503 13:59:46.063159 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.064095 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.064632 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:46.065274 12970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:46.066372 12970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:46.067523 12970 hybrid_clock.cc:648] HybridClock initialized: now 1714744786067508 us; error 34 us; skew 500 ppm
I20240503 13:59:46.069491 12970 webserver.cc:460] Webserver started at http://127.0.99.190:36213/ using document root <none> and password file <none>
I20240503 13:59:46.070122 12970 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:46.070186 12970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:46.070374 12970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:46.072094 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cd0fcedd694548f8bcce64315ec09552"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.072566 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cd0fcedd694548f8bcce64315ec09552"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.075994 12970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:46.078212 12984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.079248 12970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:46.079371 12970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cd0fcedd694548f8bcce64315ec09552"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.079457 12970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:46.089262 12970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:46.089883 12970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:46.090060 12970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:46.096426 13036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42483 every 8 connection(s)
I20240503 13:59:46.096423 12970 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42483
I20240503 13:59:46.097482 12970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:46.098227 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 12970
I20240503 13:59:46.098352 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:46.100384 13037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.106164 13037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552: Bootstrap starting.
I20240503 13:59:46.108598 13037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.109357 13037 log.cc:824] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:46.111215 13037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552: No bootstrap required, opened a new log
I20240503 13:59:46.114020 13037 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fcedd694548f8bcce64315ec09552" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42483 } }
I20240503 13:59:46.114223 13037 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.114274 13037 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0fcedd694548f8bcce64315ec09552, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.114650 13037 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fcedd694548f8bcce64315ec09552" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42483 } }
I20240503 13:59:46.114758 13037 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:46.114801 13037 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:46.114881 13037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.115605 13037 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fcedd694548f8bcce64315ec09552" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42483 } }
I20240503 13:59:46.115967 13037 leader_election.cc:302] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd0fcedd694548f8bcce64315ec09552; no voters:
I20240503 13:59:46.116297 13037 leader_election.cc:288] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:46.116637 13042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:46.116840 13042 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [term 1 LEADER]: Becoming Leader. State: Replica: cd0fcedd694548f8bcce64315ec09552, State: Running, Role: LEADER
I20240503 13:59:46.117149 13042 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fcedd694548f8bcce64315ec09552" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42483 } }
I20240503 13:59:46.117484 13037 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:46.118743 13043 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd0fcedd694548f8bcce64315ec09552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fcedd694548f8bcce64315ec09552" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42483 } } }
I20240503 13:59:46.118758 13044 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd0fcedd694548f8bcce64315ec09552. Latest consensus state: current_term: 1 leader_uuid: "cd0fcedd694548f8bcce64315ec09552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0fcedd694548f8bcce64315ec09552" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42483 } } }
I20240503 13:59:46.118866 13043 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:46.118871 13044 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:46.119194 13051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:46.122321 13051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:46.127830 13051 catalog_manager.cc:1347] Generated new cluster ID: 83409494eaf340939e4f2fab34172229
I20240503 13:59:46.127934 13051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:46.146590 13051 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:46.147965 13051 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:46.159199 13051 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cd0fcedd694548f8bcce64315ec09552: Generated new TSK 0
I20240503 13:59:46.160161 13051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:46.172693 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:46.288688 13061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:46.288934 13061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:46.288985 13061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:46.292232 13061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:46.292320 13061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:46.292414 13061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:46.296299 13061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--never_fsync=true
--heap_profile_path=/tmp/kudu.13061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:46.297335 13061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:46.298645 13061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:46.307785 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.308569 13066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.309123 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:46.310007 13061 server_base.cc:1034] running on GCE node
I20240503 13:59:46.310479 13061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:46.311136 13061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:46.312295 13061 hybrid_clock.cc:648] HybridClock initialized: now 1714744786312282 us; error 40 us; skew 500 ppm
I20240503 13:59:46.314404 13061 webserver.cc:460] Webserver started at http://127.0.99.129:40431/ using document root <none> and password file <none>
I20240503 13:59:46.315002 13061 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:46.315062 13061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:46.315243 13061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:46.316984 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.317478 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.321133 13061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:59:46.323601 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.324698 13061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.324826 13061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.324916 13061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:46.341454 13061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:46.342180 13061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:46.342356 13061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:46.342916 13061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:46.343853 13061 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:46.343904 13061 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.343958 13061 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:46.343986 13061 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.352275 13061 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33165
I20240503 13:59:46.352306 13188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33165 every 8 connection(s)
I20240503 13:59:46.353272 13061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:46.359194 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 13061
I20240503 13:59:46.359357 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:46.361704 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:46.362480 13189 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42483
I20240503 13:59:46.362725 13189 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:46.363306 13189 heartbeater.cc:511] Master 127.0.99.190:42483 requested a full tablet report, sending...
I20240503 13:59:46.364892 13001 ts_manager.cc:194] Registered new tserver with Master: 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165)
I20240503 13:59:46.366608 13001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:45755
W20240503 13:59:46.472266 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:46.472527 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:46.472573 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:46.475682 13192 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:46.475759 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:46.475828 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:46.479547 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:46.480509 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:46.481660 13192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:46.490458 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.490911 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.491784 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:46.492444 13192 server_base.cc:1034] running on GCE node
I20240503 13:59:46.492887 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:46.493525 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:46.494673 13192 hybrid_clock.cc:648] HybridClock initialized: now 1714744786494654 us; error 37 us; skew 500 ppm
I20240503 13:59:46.496660 13192 webserver.cc:460] Webserver started at http://127.0.99.130:44999/ using document root <none> and password file <none>
I20240503 13:59:46.497262 13192 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:46.497328 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:46.497510 13192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:46.499253 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ca7699ae05144c29b50003939e975810"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.499742 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ca7699ae05144c29b50003939e975810"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.503115 13192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:46.505415 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.506489 13192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:46.506595 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ca7699ae05144c29b50003939e975810"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.506682 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:46.524412 13192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:46.525059 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:46.525229 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:46.525810 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:46.526768 13192 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:46.526819 13192 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.526871 13192 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:46.526896 13192 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.534996 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:45279
I20240503 13:59:46.535017 13319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:45279 every 8 connection(s)
I20240503 13:59:46.535992 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:46.537770 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 13192
I20240503 13:59:46.537894 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:46.539938 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:46.544832 13320 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42483
I20240503 13:59:46.545107 13320 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:46.545825 13320 heartbeater.cc:511] Master 127.0.99.190:42483 requested a full tablet report, sending...
I20240503 13:59:46.546952 13001 ts_manager.cc:194] Registered new tserver with Master: ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:46.547523 13001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:43645
W20240503 13:59:46.652338 13323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:46.652596 13323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:46.652655 13323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:46.655851 13323 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:46.655937 13323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:46.656026 13323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:46.659946 13323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--never_fsync=true
--heap_profile_path=/tmp/kudu.13323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:46.660949 13323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:46.662178 13323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:46.671173 13329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.671906 13328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.672612 13331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:46.673461 13323 server_base.cc:1034] running on GCE node
I20240503 13:59:46.673965 13323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:46.674688 13323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:46.675879 13323 hybrid_clock.cc:648] HybridClock initialized: now 1714744786675855 us; error 42 us; skew 500 ppm
I20240503 13:59:46.678020 13323 webserver.cc:460] Webserver started at http://127.0.99.131:45879/ using document root <none> and password file <none>
I20240503 13:59:46.678619 13323 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:46.678683 13323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:46.678872 13323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:46.680609 13323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1ae50d36df0a4738a6e49318c46a80aa"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.681113 13323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1ae50d36df0a4738a6e49318c46a80aa"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.684792 13323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:46.687228 13337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.688380 13323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.688503 13323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1ae50d36df0a4738a6e49318c46a80aa"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.688606 13323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:46.708355 13323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:46.709082 13323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:46.709266 13323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:46.709854 13323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:46.710884 13323 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:46.710932 13323 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.710980 13323 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:46.711009 13323 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.719714 13323 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41827
I20240503 13:59:46.719727 13450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41827 every 8 connection(s)
I20240503 13:59:46.720750 13323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:46.725829 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 13323
I20240503 13:59:46.725946 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:46.728189 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:46.731418 13451 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42483
I20240503 13:59:46.731715 13451 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:46.732492 13451 heartbeater.cc:511] Master 127.0.99.190:42483 requested a full tablet report, sending...
I20240503 13:59:46.733724 13001 ts_manager.cc:194] Registered new tserver with Master: 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131:41827)
I20240503 13:59:46.734541 13001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47375
W20240503 13:59:46.844332 13454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:46.844599 13454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:46.844642 13454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:46.847790 13454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:46.847867 13454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:46.847970 13454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:46.851804 13454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42483
--never_fsync=true
--heap_profile_path=/tmp/kudu.13454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:46.852831 13454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:46.854089 13454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:46.863656 13459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.864336 13462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:46.864881 13460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:46.865726 13454 server_base.cc:1034] running on GCE node
I20240503 13:59:46.866204 13454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:46.866853 13454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:46.868034 13454 hybrid_clock.cc:648] HybridClock initialized: now 1714744786868003 us; error 47 us; skew 500 ppm
I20240503 13:59:46.870241 13454 webserver.cc:460] Webserver started at http://127.0.99.132:35101/ using document root <none> and password file <none>
I20240503 13:59:46.870884 13454 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:46.870945 13454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:46.871140 13454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:46.872915 13454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.873402 13454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.877125 13454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:59:46.879618 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.880731 13454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:46.880856 13454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:46.880970 13454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:46.892297 13454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:46.892962 13454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:46.893132 13454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:46.893723 13454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:46.894726 13454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:46.894776 13454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.894821 13454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:46.894872 13454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:46.903584 13454 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34417
I20240503 13:59:46.903599 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34417 every 8 connection(s)
I20240503 13:59:46.904677 13454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:46.913822 13582 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42483
I20240503 13:59:46.914095 13582 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:46.914631 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 13454
I20240503 13:59:46.914678 13582 heartbeater.cc:511] Master 127.0.99.190:42483 requested a full tablet report, sending...
I20240503 13:59:46.914757 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:46.915869 13000 ts_manager.cc:194] Registered new tserver with Master: 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132:34417)
I20240503 13:59:46.916528 13000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:48167
I20240503 13:59:46.917055 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 13:59:46.926640 13001 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38022:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:59:46.943670 13385 tablet_service.cc:1467] Processing CreateTablet for tablet 636cea461e514757a78ea545202d5148 (DEFAULT_TABLE table=TestTable [id=83a793fece714092bdc5a5149c924861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:46.944728 13385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636cea461e514757a78ea545202d5148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:46.950660 13595 tablet_bootstrap.cc:492] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap starting.
I20240503 13:59:46.952716 13595 tablet_bootstrap.cc:654] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:46.953461 13595 log.cc:824] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:46.955293 13595 tablet_bootstrap.cc:492] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: No bootstrap required, opened a new log
I20240503 13:59:46.955493 13595 ts_tablet_manager.cc:1367] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:46.958581 13595 raft_consensus.cc:348] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:46.958815 13595 raft_consensus.cc:374] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:46.958892 13595 raft_consensus.cc:729] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Initialized, Role: FOLLOWER
I20240503 13:59:46.959344 13595 consensus_queue.cc:260] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:46.959471 13595 raft_consensus.cc:388] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:46.959535 13595 raft_consensus.cc:482] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:46.959627 13595 raft_consensus.cc:3037] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:46.960602 13595 raft_consensus.cc:504] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:46.960969 13595 leader_election.cc:302] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae50d36df0a4738a6e49318c46a80aa; no voters:
I20240503 13:59:46.961277 13595 leader_election.cc:288] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:46.961403 13597 raft_consensus.cc:2781] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:46.961634 13597 raft_consensus.cc:686] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEADER]: Becoming Leader. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Running, Role: LEADER
I20240503 13:59:46.961903 13451 heartbeater.cc:503] Master 127.0.99.190:42483 was elected leader, sending a full tablet report...
I20240503 13:59:46.962004 13597 consensus_queue.cc:237] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:46.962164 13595 ts_tablet_manager.cc:1398] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 13:59:46.965795 13001 catalog_manager.cc:5484] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131). New cstate: current_term: 1 leader_uuid: "1ae50d36df0a4738a6e49318c46a80aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:46.979619 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 13:59:46.981031 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5008569cd3fd47ac8c2f3d2879e3d3f1 to finish bootstrapping
I20240503 13:59:46.987205 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ca7699ae05144c29b50003939e975810 to finish bootstrapping
I20240503 13:59:46.993278 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1ae50d36df0a4738a6e49318c46a80aa to finish bootstrapping
I20240503 13:59:46.998324 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 91f8d9c045af4db4a6a3d4d39e5a55e8 to finish bootstrapping
I20240503 13:59:47.004190 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 13323
I20240503 13:59:47.009497 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 13454
I20240503 13:59:47.369232 13189 heartbeater.cc:503] Master 127.0.99.190:42483 was elected leader, sending a full tablet report...
I20240503 13:59:47.550011 13320 heartbeater.cc:503] Master 127.0.99.190:42483 was elected leader, sending a full tablet report...
I20240503 13:59:50.765782 13001 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38022:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:50.779244 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd37b00b3ee49b1bdda91019af39c57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=914a6956449240e1afba80f78f23c7a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:50.779242 13122 tablet_service.cc:1467] Processing CreateTablet for tablet 46aad7ddf91b4098b9d4ec0c862f3173 (DEFAULT_TABLE table=rebalance_test_table_0 [id=914a6956449240e1afba80f78f23c7a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:50.779253 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 558c698a762b4742971f25f5d997013c (DEFAULT_TABLE table=rebalance_test_table_0 [id=914a6956449240e1afba80f78f23c7a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:50.780258 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c698a762b4742971f25f5d997013c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.780287 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd37b00b3ee49b1bdda91019af39c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.780576 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46aad7ddf91b4098b9d4ec0c862f3173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.785945 13614 tablet_bootstrap.cc:492] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1: Bootstrap starting.
I20240503 13:59:50.786264 13615 tablet_bootstrap.cc:492] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810: Bootstrap starting.
I20240503 13:59:50.788079 13614 tablet_bootstrap.cc:654] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:50.788100 13615 tablet_bootstrap.cc:654] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:50.788753 13614 log.cc:824] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:50.788753 13615 log.cc:824] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:50.790315 13615 tablet_bootstrap.cc:492] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810: No bootstrap required, opened a new log
I20240503 13:59:50.790314 13614 tablet_bootstrap.cc:492] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1: No bootstrap required, opened a new log
I20240503 13:59:50.790489 13614 ts_tablet_manager.cc:1367] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 13:59:50.790489 13615 ts_tablet_manager.cc:1367] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 13:59:50.793565 13614 raft_consensus.cc:348] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.793565 13615 raft_consensus.cc:348] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:50.793758 13614 raft_consensus.cc:374] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:50.793759 13615 raft_consensus.cc:374] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:50.793792 13614 raft_consensus.cc:729] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Initialized, Role: FOLLOWER
I20240503 13:59:50.793810 13615 raft_consensus.cc:729] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Initialized, Role: FOLLOWER
I20240503 13:59:50.794229 13615 consensus_queue.cc:260] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:50.794229 13614 consensus_queue.cc:260] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.794353 13615 raft_consensus.cc:388] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:50.794349 13614 raft_consensus.cc:388] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:50.794412 13614 raft_consensus.cc:482] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:50.794413 13615 raft_consensus.cc:482] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:50.794512 13615 raft_consensus.cc:3037] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:50.794513 13614 raft_consensus.cc:3037] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:50.795305 13615 raft_consensus.cc:504] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:50.795320 13614 raft_consensus.cc:504] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.795625 13614 leader_election.cc:302] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1; no voters:
I20240503 13:59:50.795625 13615 leader_election.cc:302] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:50.795920 13615 leader_election.cc:288] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:50.795920 13614 leader_election.cc:288] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:50.796015 13618 raft_consensus.cc:2781] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:50.796172 13618 raft_consensus.cc:686] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:50.796265 13621 raft_consensus.cc:2781] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:50.796468 13621 raft_consensus.cc:686] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Leader. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:50.796458 13618 consensus_queue.cc:237] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.796779 13614 ts_tablet_manager.cc:1398] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240503 13:59:50.796782 13615 ts_tablet_manager.cc:1398] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:59:50.796814 13621 consensus_queue.cc:237] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:50.797161 13614 tablet_bootstrap.cc:492] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Bootstrap starting.
I20240503 13:59:50.798238 13614 tablet_bootstrap.cc:654] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:50.799327 13001 catalog_manager.cc:5484] T 558c698a762b4742971f25f5d997013c P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:50.801340 13614 tablet_bootstrap.cc:492] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: No bootstrap required, opened a new log
I20240503 13:59:50.801452 13614 ts_tablet_manager.cc:1367] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:50.801851 13000 catalog_manager.cc:5484] T 8bd37b00b3ee49b1bdda91019af39c57 P ca7699ae05144c29b50003939e975810 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7699ae05144c29b50003939e975810 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:50.802054 13614 raft_consensus.cc:348] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.802170 13614 raft_consensus.cc:374] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:50.802209 13614 raft_consensus.cc:729] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Initialized, Role: FOLLOWER
I20240503 13:59:50.802336 13614 consensus_queue.cc:260] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.802415 13614 raft_consensus.cc:388] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:50.802450 13614 raft_consensus.cc:482] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:50.802488 13614 raft_consensus.cc:3037] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:50.803339 13614 raft_consensus.cc:504] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.803498 13614 leader_election.cc:302] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1; no voters:
I20240503 13:59:50.803591 13614 leader_election.cc:288] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:50.803655 13620 raft_consensus.cc:2781] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:50.803773 13620 raft_consensus.cc:686] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:50.803885 13620 consensus_queue.cc:237] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:50.804082 13614 ts_tablet_manager.cc:1398] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:50.805547 13001 catalog_manager.cc:5484] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:50.993592 13001 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38022:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:50.999106 13254 tablet_service.cc:1467] Processing CreateTablet for tablet f54dc5537468409c9fe2117431d58634 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76fbe485b57a437cab200393da94cf52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:50.999130 13253 tablet_service.cc:1467] Processing CreateTablet for tablet 1c43637c1afe4ec5b01e5dda46ed9086 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76fbe485b57a437cab200393da94cf52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:50.999285 13122 tablet_service.cc:1467] Processing CreateTablet for tablet e6db7b0e734c4165abe1a4b3acc1511e (DEFAULT_TABLE table=rebalance_test_table_1 [id=76fbe485b57a437cab200393da94cf52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:50.999486 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54dc5537468409c9fe2117431d58634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.999558 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6db7b0e734c4165abe1a4b3acc1511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:50.999642 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c43637c1afe4ec5b01e5dda46ed9086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.001189 13615 tablet_bootstrap.cc:492] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Bootstrap starting.
I20240503 13:59:51.001610 13614 tablet_bootstrap.cc:492] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Bootstrap starting.
I20240503 13:59:51.002162 13615 tablet_bootstrap.cc:654] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.002504 13614 tablet_bootstrap.cc:654] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.003324 13615 tablet_bootstrap.cc:492] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: No bootstrap required, opened a new log
I20240503 13:59:51.003407 13615 ts_tablet_manager.cc:1367] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:51.003746 13614 tablet_bootstrap.cc:492] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1: No bootstrap required, opened a new log
I20240503 13:59:51.003831 13614 ts_tablet_manager.cc:1367] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:51.004011 13615 raft_consensus.cc:348] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.004115 13615 raft_consensus.cc:374] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.004154 13615 raft_consensus.cc:729] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.004256 13614 raft_consensus.cc:348] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.004288 13615 consensus_queue.cc:260] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.004348 13614 raft_consensus.cc:374] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.004362 13615 raft_consensus.cc:388] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.004386 13614 raft_consensus.cc:729] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.004413 13615 raft_consensus.cc:482] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.004460 13615 raft_consensus.cc:3037] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.004514 13614 consensus_queue.cc:260] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.004598 13614 raft_consensus.cc:388] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.004634 13614 raft_consensus.cc:482] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.004678 13614 raft_consensus.cc:3037] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.005296 13615 raft_consensus.cc:504] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.005445 13615 leader_election.cc:302] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:51.005479 13614 raft_consensus.cc:504] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.005549 13615 leader_election.cc:288] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.005600 13614 leader_election.cc:302] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1; no voters:
I20240503 13:59:51.005616 13623 raft_consensus.cc:2781] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.005699 13623 raft_consensus.cc:686] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Leader. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:51.005723 13620 raft_consensus.cc:2781] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.005801 13620 raft_consensus.cc:686] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:51.005810 13623 consensus_queue.cc:237] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.005904 13615 ts_tablet_manager.cc:1398] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:51.005914 13620 consensus_queue.cc:237] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.006107 13615 tablet_bootstrap.cc:492] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810: Bootstrap starting.
I20240503 13:59:51.006350 13614 leader_election.cc:288] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.006614 13614 ts_tablet_manager.cc:1398] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:51.007006 13615 tablet_bootstrap.cc:654] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.007568 13001 catalog_manager.cc:5484] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7699ae05144c29b50003939e975810 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.007850 13000 catalog_manager.cc:5484] T e6db7b0e734c4165abe1a4b3acc1511e P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.008265 13615 tablet_bootstrap.cc:492] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810: No bootstrap required, opened a new log
I20240503 13:59:51.008332 13615 ts_tablet_manager.cc:1367] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:51.008982 13615 raft_consensus.cc:348] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.009068 13615 raft_consensus.cc:374] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.009096 13615 raft_consensus.cc:729] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.009207 13615 consensus_queue.cc:260] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.009279 13615 raft_consensus.cc:388] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.009310 13615 raft_consensus.cc:482] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.009344 13615 raft_consensus.cc:3037] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.010221 13615 raft_consensus.cc:504] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.010344 13615 leader_election.cc:302] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:51.010439 13615 leader_election.cc:288] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.010496 13619 raft_consensus.cc:2781] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.010577 13619 raft_consensus.cc:686] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Leader. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:51.010633 13615 ts_tablet_manager.cc:1398] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:51.010685 13619 consensus_queue.cc:237] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.012668 12999 catalog_manager.cc:5484] T 1c43637c1afe4ec5b01e5dda46ed9086 P ca7699ae05144c29b50003939e975810 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7699ae05144c29b50003939e975810 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.190138 12999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38022:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:51.195950 13253 tablet_service.cc:1467] Processing CreateTablet for tablet 661cf4663eb04784bd3336c626397d00 (DEFAULT_TABLE table=rebalance_test_table_2 [id=18d432015e4149348a15b645e6b8697c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:51.195937 13122 tablet_service.cc:1467] Processing CreateTablet for tablet 94180255c91e4ff49d20f0fef0c71c4b (DEFAULT_TABLE table=rebalance_test_table_2 [id=18d432015e4149348a15b645e6b8697c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:51.195955 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 60986ef8847e491ca968d60a5c8d302e (DEFAULT_TABLE table=rebalance_test_table_2 [id=18d432015e4149348a15b645e6b8697c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:51.196269 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661cf4663eb04784bd3336c626397d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.196265 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94180255c91e4ff49d20f0fef0c71c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.196486 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60986ef8847e491ca968d60a5c8d302e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.198027 13615 tablet_bootstrap.cc:492] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Bootstrap starting.
I20240503 13:59:51.198449 13614 tablet_bootstrap.cc:492] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1: Bootstrap starting.
I20240503 13:59:51.199039 13615 tablet_bootstrap.cc:654] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.199348 13614 tablet_bootstrap.cc:654] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.200183 13615 tablet_bootstrap.cc:492] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: No bootstrap required, opened a new log
I20240503 13:59:51.200268 13615 ts_tablet_manager.cc:1367] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.200325 13614 tablet_bootstrap.cc:492] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1: No bootstrap required, opened a new log
I20240503 13:59:51.200410 13614 ts_tablet_manager.cc:1367] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:51.200685 13615 raft_consensus.cc:348] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.200778 13615 raft_consensus.cc:374] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.200807 13615 raft_consensus.cc:729] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.200793 13614 raft_consensus.cc:348] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.200874 13614 raft_consensus.cc:374] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.200919 13614 raft_consensus.cc:729] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.200946 13615 consensus_queue.cc:260] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.201035 13615 raft_consensus.cc:388] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.201028 13614 consensus_queue.cc:260] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.201077 13615 raft_consensus.cc:482] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.201105 13614 raft_consensus.cc:388] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.201121 13615 raft_consensus.cc:3037] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.201143 13614 raft_consensus.cc:482] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.201187 13614 raft_consensus.cc:3037] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.201934 13615 raft_consensus.cc:504] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.201951 13614 raft_consensus.cc:504] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.202085 13615 leader_election.cc:302] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:51.202102 13614 leader_election.cc:302] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1; no voters:
I20240503 13:59:51.202188 13614 leader_election.cc:288] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.202203 13619 raft_consensus.cc:2781] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.202270 13619 raft_consensus.cc:686] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Leader. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:51.202405 13614 ts_tablet_manager.cc:1398] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.202389 13619 consensus_queue.cc:237] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.202584 13614 tablet_bootstrap.cc:492] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Bootstrap starting.
I20240503 13:59:51.203491 13614 tablet_bootstrap.cc:654] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.203539 13615 leader_election.cc:288] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.203792 13615 ts_tablet_manager.cc:1398] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:51.204082 13620 raft_consensus.cc:2781] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.204169 13620 raft_consensus.cc:686] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:51.204291 13620 consensus_queue.cc:237] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.204470 12999 catalog_manager.cc:5484] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7699ae05144c29b50003939e975810 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.205832 13614 tablet_bootstrap.cc:492] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: No bootstrap required, opened a new log
I20240503 13:59:51.205770 13001 catalog_manager.cc:5484] T 94180255c91e4ff49d20f0fef0c71c4b P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.205924 13614 ts_tablet_manager.cc:1367] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:51.206393 13614 raft_consensus.cc:348] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.206492 13614 raft_consensus.cc:374] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.206530 13614 raft_consensus.cc:729] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.206640 13614 consensus_queue.cc:260] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.206722 13614 raft_consensus.cc:388] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.206756 13614 raft_consensus.cc:482] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.206796 13614 raft_consensus.cc:3037] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.207737 13614 raft_consensus.cc:504] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.207867 13614 leader_election.cc:302] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1; no voters:
I20240503 13:59:51.208009 13614 leader_election.cc:288] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.208055 13618 raft_consensus.cc:2781] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.208153 13618 raft_consensus.cc:686] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:51.208261 13618 consensus_queue.cc:237] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.208328 13614 ts_tablet_manager.cc:1398] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.210240 12999 catalog_manager.cc:5484] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.381726 12999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38022:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:51.387499 13253 tablet_service.cc:1467] Processing CreateTablet for tablet f4844a0937c7491e955ee51beceada38 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448a9a630f3e460a824c802b0b1ff382]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:51.387574 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8af25de83048a48ba662dbbd6b551e (DEFAULT_TABLE table=rebalance_test_table_3 [id=448a9a630f3e460a824c802b0b1ff382]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:51.387499 13254 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce20279e6b41d5a527ba472422314f (DEFAULT_TABLE table=rebalance_test_table_3 [id=448a9a630f3e460a824c802b0b1ff382]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:51.387883 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8af25de83048a48ba662dbbd6b551e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.387907 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce20279e6b41d5a527ba472422314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.390072 13615 tablet_bootstrap.cc:492] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Bootstrap starting.
I20240503 13:59:51.390074 13614 tablet_bootstrap.cc:492] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Bootstrap starting.
I20240503 13:59:51.390307 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4844a0937c7491e955ee51beceada38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:51.390981 13615 tablet_bootstrap.cc:654] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.390981 13614 tablet_bootstrap.cc:654] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.392345 13615 tablet_bootstrap.cc:492] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: No bootstrap required, opened a new log
I20240503 13:59:51.392431 13614 tablet_bootstrap.cc:492] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1: No bootstrap required, opened a new log
I20240503 13:59:51.392442 13615 ts_tablet_manager.cc:1367] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.392519 13614 ts_tablet_manager.cc:1367] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.392952 13614 raft_consensus.cc:348] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.392937 13615 raft_consensus.cc:348] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.393049 13614 raft_consensus.cc:374] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.393049 13615 raft_consensus.cc:374] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.393087 13615 raft_consensus.cc:729] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.393086 13614 raft_consensus.cc:729] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.393219 13615 consensus_queue.cc:260] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.393219 13614 consensus_queue.cc:260] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.393301 13614 raft_consensus.cc:388] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.393301 13615 raft_consensus.cc:388] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.393344 13615 raft_consensus.cc:482] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.393342 13614 raft_consensus.cc:482] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.393393 13615 raft_consensus.cc:3037] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.393393 13614 raft_consensus.cc:3037] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.394264 13615 raft_consensus.cc:504] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.394264 13614 raft_consensus.cc:504] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.394428 13615 leader_election.cc:302] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:51.394464 13614 leader_election.cc:302] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1; no voters:
I20240503 13:59:51.394534 13615 leader_election.cc:288] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.394593 13614 leader_election.cc:288] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.394598 13623 raft_consensus.cc:2781] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.394654 13618 raft_consensus.cc:2781] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.394687 13623 raft_consensus.cc:686] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Leader. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:51.394732 13618 raft_consensus.cc:686] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:51.394770 13615 ts_tablet_manager.cc:1398] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:51.394829 13618 consensus_queue.cc:237] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } }
I20240503 13:59:51.394809 13623 consensus_queue.cc:237] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.396363 13615 tablet_bootstrap.cc:492] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810: Bootstrap starting.
I20240503 13:59:51.397003 13001 catalog_manager.cc:5484] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7699ae05144c29b50003939e975810 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.397230 13000 catalog_manager.cc:5484] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.398677 13615 tablet_bootstrap.cc:654] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:51.400318 13614 ts_tablet_manager.cc:1398] T 4e8af25de83048a48ba662dbbd6b551e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240503 13:59:51.401568 13615 tablet_bootstrap.cc:492] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810: No bootstrap required, opened a new log
I20240503 13:59:51.401671 13615 ts_tablet_manager.cc:1367] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:51.402189 13615 raft_consensus.cc:348] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.402318 13615 raft_consensus.cc:374] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:51.402364 13615 raft_consensus.cc:729] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.402523 13615 consensus_queue.cc:260] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.402624 13615 raft_consensus.cc:388] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.402662 13615 raft_consensus.cc:482] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.402698 13615 raft_consensus.cc:3037] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:51.403488 13615 raft_consensus.cc:504] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.403628 13615 leader_election.cc:302] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:51.403738 13615 leader_election.cc:288] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:51.403796 13623 raft_consensus.cc:2781] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:51.403906 13623 raft_consensus.cc:686] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Leader. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:51.404011 13623 consensus_queue.cc:237] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } }
I20240503 13:59:51.404103 13615 ts_tablet_manager.cc:1398] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:51.405251 13000 catalog_manager.cc:5484] T f4844a0937c7491e955ee51beceada38 P ca7699ae05144c29b50003939e975810 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7699ae05144c29b50003939e975810 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:51.583352 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:41827
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:42483
--webserver_port=45879
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:51.696992 13720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:51.697244 13720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:51.697289 13720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:51.700486 13720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:51.700562 13720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:51.700634 13720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:51.704471 13720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:41827
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=45879
--tserver_master_addrs=127.0.99.190:42483
--never_fsync=true
--heap_profile_path=/tmp/kudu.13720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:51.705417 13720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:51.706609 13720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:51.715922 13728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:51.716390 13726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:51.717257 13725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:51.717949 13720 server_base.cc:1034] running on GCE node
I20240503 13:59:51.718474 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:51.719139 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:51.720280 13720 hybrid_clock.cc:648] HybridClock initialized: now 1714744791720246 us; error 52 us; skew 500 ppm
I20240503 13:59:51.722802 13720 webserver.cc:460] Webserver started at http://127.0.99.131:45879/ using document root <none> and password file <none>
I20240503 13:59:51.723629 13720 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:51.723721 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:51.728328 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 13:59:51.731037 13734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:51.732115 13720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.732247 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1ae50d36df0a4738a6e49318c46a80aa"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:51.732684 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:51.753518 13720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:51.754328 13720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:51.754514 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:51.755136 13720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:51.756453 13741 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 13:59:51.759677 13720 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 13:59:51.759760 13720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:51.759830 13720 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 13:59:51.761387 13720 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 13:59:51.761431 13720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 13:59:51.761685 13741 tablet_bootstrap.cc:492] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap starting.
I20240503 13:59:51.771142 13720 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41827
I20240503 13:59:51.771173 13848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41827 every 8 connection(s)
I20240503 13:59:51.772305 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:51.776105 13741 log.cc:824] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:51.778489 13741 tablet_bootstrap.cc:492] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:51.778829 13741 tablet_bootstrap.cc:492] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap complete.
I20240503 13:59:51.779438 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 13720
I20240503 13:59:51.779896 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:34417
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:42483
--webserver_port=35101
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:51.779937 13741 ts_tablet_manager.cc:1367] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.009s
I20240503 13:59:51.781035 13849 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42483
I20240503 13:59:51.781239 13849 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:51.781924 13849 heartbeater.cc:511] Master 127.0.99.190:42483 requested a full tablet report, sending...
I20240503 13:59:51.783134 13741 raft_consensus.cc:348] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:51.783565 13741 raft_consensus.cc:729] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Initialized, Role: FOLLOWER
I20240503 13:59:51.783885 13000 ts_manager.cc:194] Re-registered known tserver with Master: 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131:41827)
I20240503 13:59:51.784142 13741 consensus_queue.cc:260] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:51.784299 13741 raft_consensus.cc:388] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:51.784375 13741 raft_consensus.cc:482] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:51.784489 13741 raft_consensus.cc:3037] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:51.784741 13000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:54031
I20240503 13:59:51.786721 13741 raft_consensus.cc:504] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:51.787127 13741 leader_election.cc:302] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae50d36df0a4738a6e49318c46a80aa; no voters:
I20240503 13:59:51.787463 13741 leader_election.cc:288] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 13:59:51.787566 13854 raft_consensus.cc:2781] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:51.787762 13854 raft_consensus.cc:686] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 LEADER]: Becoming Leader. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Running, Role: LEADER
I20240503 13:59:51.788132 13849 heartbeater.cc:503] Master 127.0.99.190:42483 was elected leader, sending a full tablet report...
I20240503 13:59:51.788090 13854 consensus_queue.cc:237] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } }
I20240503 13:59:51.788431 13741 ts_tablet_manager.cc:1398] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
I20240503 13:59:51.790385 13000 catalog_manager.cc:5484] T 636cea461e514757a78ea545202d5148 P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "1ae50d36df0a4738a6e49318c46a80aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:51.894510 13853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:51.894781 13853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:51.894837 13853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:51.898234 13853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:51.898321 13853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:51.898401 13853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:51.902321 13853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:34417
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=35101
--tserver_master_addrs=127.0.99.190:42483
--never_fsync=true
--heap_profile_path=/tmp/kudu.13853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:51.903357 13853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:51.904647 13853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:51.914366 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:51.915172 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:51.915659 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:51.916540 13853 server_base.cc:1034] running on GCE node
I20240503 13:59:51.917014 13853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:51.917732 13853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:51.918926 13853 hybrid_clock.cc:648] HybridClock initialized: now 1714744791918872 us; error 66 us; skew 500 ppm
I20240503 13:59:51.921451 13853 webserver.cc:460] Webserver started at http://127.0.99.132:35101/ using document root <none> and password file <none>
I20240503 13:59:51.922134 13853 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:51.922248 13853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:51.926522 13853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:51.928798 13872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:51.930016 13853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:51.930140 13853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
format_stamp: "Formatted at 2024-05-03 13:59:46 on dist-test-slave-6fwv"
I20240503 13:59:51.930599 13853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:51.968490 13853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:51.969197 13853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:51.969372 13853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:51.970019 13853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:51.970983 13853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:51.971035 13853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:51.971076 13853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:51.971096 13853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:51.979765 13853 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34417
I20240503 13:59:51.979796 13985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34417 every 8 connection(s)
I20240503 13:59:51.980939 13853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:51.987344 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 13853
I20240503 13:59:51.990888 13986 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42483
I20240503 13:59:51.991151 13986 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:51.991750 13986 heartbeater.cc:511] Master 127.0.99.190:42483 requested a full tablet report, sending...
I20240503 13:59:51.992899 13000 ts_manager.cc:194] Re-registered known tserver with Master: 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132:34417)
I20240503 13:59:51.993678 13000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:48905
I20240503 13:59:52.187922 13123 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:52.188501 13253 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:52.190068 13783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:52.193733 13920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:52.291425 13123 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:52.291833 13920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:52.292520 13783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:52.292692 13253 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:52.403105 13783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:52.404175 13123 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:52.404980 13920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:52.406287 13253 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:52.446049 13143 consensus_queue.cc:237] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.447587 14103 raft_consensus.cc:2932] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } } }
I20240503 13:59:52.450273 12999 catalog_manager.cc:5484] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:52.452832 13076 consensus_peers.cc:485] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 -> Peer 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132:34417): Couldn't send request to peer 91f8d9c045af4db4a6a3d4d39e5a55e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60986ef8847e491ca968d60a5c8d302e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.459396 13274 consensus_queue.cc:237] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } }
I20240503 13:59:52.460925 14116 raft_consensus.cc:2932] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } } }
I20240503 13:59:52.463752 12999 catalog_manager.cc:5484] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:52.466890 13209 consensus_peers.cc:485] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 -> Peer 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131:41827): Couldn't send request to peer 1ae50d36df0a4738a6e49318c46a80aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3ce20279e6b41d5a527ba472422314f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.471454 13143 consensus_queue.cc:237] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.472400 14103 raft_consensus.cc:2932] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } } }
W20240503 13:59:52.472834 13076 consensus_peers.cc:485] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 -> Peer 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132:34417): Couldn't send request to peer 91f8d9c045af4db4a6a3d4d39e5a55e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46aad7ddf91b4098b9d4ec0c862f3173. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.474932 12999 catalog_manager.cc:5484] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.481648 13274 consensus_queue.cc:237] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } }
I20240503 13:59:52.482955 14116 raft_consensus.cc:2932] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } } }
W20240503 13:59:52.483306 13209 consensus_peers.cc:485] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 -> Peer 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131:41827): Couldn't send request to peer 1ae50d36df0a4738a6e49318c46a80aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 661cf4663eb04784bd3336c626397d00. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.485297 12999 catalog_manager.cc:5484] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:52.492205 13274 consensus_queue.cc:237] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.493181 14131 raft_consensus.cc:2932] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } } }
I20240503 13:59:52.495757 12999 catalog_manager.cc:5484] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 reported cstate change: config changed from index -1 to 3, NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:52.497005 13207 consensus_peers.cc:485] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 -> Peer 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132:34417): Couldn't send request to peer 91f8d9c045af4db4a6a3d4d39e5a55e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f54dc5537468409c9fe2117431d58634. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:52.864427 14188 ts_tablet_manager.cc:902] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Initiating tablet copy from peer 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165)
I20240503 13:59:52.865164 14188 tablet_copy_client.cc:308] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33165
I20240503 13:59:52.869069 13163 tablet_copy_service.cc:140] P 5008569cd3fd47ac8c2f3d2879e3d3f1: Received BeginTabletCopySession request for tablet 60986ef8847e491ca968d60a5c8d302e from peer 91f8d9c045af4db4a6a3d4d39e5a55e8 ({username='slave'} at 127.0.99.132:57749)
I20240503 13:59:52.869236 13163 tablet_copy_service.cc:161] P 5008569cd3fd47ac8c2f3d2879e3d3f1: Beginning new tablet copy session on tablet 60986ef8847e491ca968d60a5c8d302e from peer 91f8d9c045af4db4a6a3d4d39e5a55e8 at {username='slave'} at 127.0.99.132:57749: session id = 91f8d9c045af4db4a6a3d4d39e5a55e8-60986ef8847e491ca968d60a5c8d302e
I20240503 13:59:52.870550 13163 tablet_copy_source_session.cc:205] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:52.874008 14188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60986ef8847e491ca968d60a5c8d302e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:52.877537 14194 ts_tablet_manager.cc:902] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Initiating tablet copy from peer ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:52.878086 14194 tablet_copy_client.cc:308] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:45279
I20240503 13:59:52.880353 14188 tablet_copy_client.cc:783] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:52.880564 14188 tablet_copy_client.cc:647] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:52.882105 13294 tablet_copy_service.cc:140] P ca7699ae05144c29b50003939e975810: Received BeginTabletCopySession request for tablet f54dc5537468409c9fe2117431d58634 from peer 91f8d9c045af4db4a6a3d4d39e5a55e8 ({username='slave'} at 127.0.99.132:40615)
I20240503 13:59:52.882264 13294 tablet_copy_service.cc:161] P ca7699ae05144c29b50003939e975810: Beginning new tablet copy session on tablet f54dc5537468409c9fe2117431d58634 from peer 91f8d9c045af4db4a6a3d4d39e5a55e8 at {username='slave'} at 127.0.99.132:40615: session id = 91f8d9c045af4db4a6a3d4d39e5a55e8-f54dc5537468409c9fe2117431d58634
I20240503 13:59:52.882553 14188 tablet_copy_client.cc:523] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:52.883766 13294 tablet_copy_source_session.cc:205] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:52.884588 14188 tablet_bootstrap.cc:492] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap starting.
I20240503 13:59:52.884928 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f54dc5537468409c9fe2117431d58634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:52.886590 14194 tablet_copy_client.cc:783] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:52.886724 14194 tablet_copy_client.cc:647] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:52.888235 14194 tablet_copy_client.cc:523] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:52.889251 14194 tablet_bootstrap.cc:492] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap starting.
I20240503 13:59:52.889715 14188 log.cc:824] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:52.892134 14188 tablet_bootstrap.cc:492] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:52.892403 14188 tablet_bootstrap.cc:492] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap complete.
I20240503 13:59:52.892668 14188 ts_tablet_manager.cc:1367] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 13:59:52.894222 14201 ts_tablet_manager.cc:902] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: Initiating tablet copy from peer ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:52.894855 14201 tablet_copy_client.cc:308] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:45279
I20240503 13:59:52.895709 14188 raft_consensus.cc:348] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.895974 14188 raft_consensus.cc:729] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91f8d9c045af4db4a6a3d4d39e5a55e8, State: Initialized, Role: LEARNER
I20240503 13:59:52.896507 14188 consensus_queue.cc:260] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.897562 14188 ts_tablet_manager.cc:1398] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20240503 13:59:52.897966 13986 heartbeater.cc:503] Master 127.0.99.190:42483 was elected leader, sending a full tablet report...
I20240503 13:59:52.898234 14194 tablet_bootstrap.cc:492] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:52.898491 14194 tablet_bootstrap.cc:492] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap complete.
I20240503 13:59:52.898452 13163 tablet_copy_service.cc:342] P 5008569cd3fd47ac8c2f3d2879e3d3f1: Request end of tablet copy session 91f8d9c045af4db4a6a3d4d39e5a55e8-60986ef8847e491ca968d60a5c8d302e received from {username='slave'} at 127.0.99.132:57749
I20240503 13:59:52.898540 13163 tablet_copy_service.cc:433] P 5008569cd3fd47ac8c2f3d2879e3d3f1: ending tablet copy session 91f8d9c045af4db4a6a3d4d39e5a55e8-60986ef8847e491ca968d60a5c8d302e on tablet 60986ef8847e491ca968d60a5c8d302e with peer 91f8d9c045af4db4a6a3d4d39e5a55e8
I20240503 13:59:52.898660 14194 ts_tablet_manager.cc:1367] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20240503 13:59:52.899264 14194 raft_consensus.cc:348] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.899356 14194 raft_consensus.cc:729] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91f8d9c045af4db4a6a3d4d39e5a55e8, State: Initialized, Role: LEARNER
I20240503 13:59:52.899438 14194 consensus_queue.cc:260] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:52.899735 14194 ts_tablet_manager.cc:1398] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:52.899854 13294 tablet_copy_service.cc:140] P ca7699ae05144c29b50003939e975810: Received BeginTabletCopySession request for tablet 661cf4663eb04784bd3336c626397d00 from peer 1ae50d36df0a4738a6e49318c46a80aa ({username='slave'} at 127.0.99.131:56027)
I20240503 13:59:52.899945 13294 tablet_copy_service.cc:161] P ca7699ae05144c29b50003939e975810: Beginning new tablet copy session on tablet 661cf4663eb04784bd3336c626397d00 from peer 1ae50d36df0a4738a6e49318c46a80aa at {username='slave'} at 127.0.99.131:56027: session id = 1ae50d36df0a4738a6e49318c46a80aa-661cf4663eb04784bd3336c626397d00
I20240503 13:59:52.900825 13294 tablet_copy_source_session.cc:205] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:52.901500 13294 tablet_copy_service.cc:342] P ca7699ae05144c29b50003939e975810: Request end of tablet copy session 91f8d9c045af4db4a6a3d4d39e5a55e8-f54dc5537468409c9fe2117431d58634 received from {username='slave'} at 127.0.99.132:40615
I20240503 13:59:52.901595 13294 tablet_copy_service.cc:433] P ca7699ae05144c29b50003939e975810: ending tablet copy session 91f8d9c045af4db4a6a3d4d39e5a55e8-f54dc5537468409c9fe2117431d58634 on tablet f54dc5537468409c9fe2117431d58634 with peer 91f8d9c045af4db4a6a3d4d39e5a55e8
I20240503 13:59:52.902727 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661cf4663eb04784bd3336c626397d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:52.905570 14201 tablet_copy_client.cc:783] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:52.905738 14201 tablet_copy_client.cc:647] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:52.907207 14201 tablet_copy_client.cc:523] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:52.908493 14201 tablet_bootstrap.cc:492] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap starting.
I20240503 13:59:52.911911 14201 tablet_bootstrap.cc:492] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:52.912147 14201 tablet_bootstrap.cc:492] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap complete.
I20240503 13:59:52.912293 14201 ts_tablet_manager.cc:1367] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:52.912811 14201 raft_consensus.cc:348] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } }
I20240503 13:59:52.912918 14201 raft_consensus.cc:729] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Initialized, Role: LEARNER
I20240503 13:59:52.913020 14201 consensus_queue.cc:260] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } }
I20240503 13:59:52.913345 14201 ts_tablet_manager.cc:1398] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:52.914052 13294 tablet_copy_service.cc:342] P ca7699ae05144c29b50003939e975810: Request end of tablet copy session 1ae50d36df0a4738a6e49318c46a80aa-661cf4663eb04784bd3336c626397d00 received from {username='slave'} at 127.0.99.131:56027
I20240503 13:59:52.914147 13294 tablet_copy_service.cc:433] P ca7699ae05144c29b50003939e975810: ending tablet copy session 1ae50d36df0a4738a6e49318c46a80aa-661cf4663eb04784bd3336c626397d00 on tablet 661cf4663eb04784bd3336c626397d00 with peer 1ae50d36df0a4738a6e49318c46a80aa
I20240503 13:59:52.931054 14201 ts_tablet_manager.cc:902] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: Initiating tablet copy from peer ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:52.931543 14201 tablet_copy_client.cc:308] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:45279
I20240503 13:59:52.932101 13294 tablet_copy_service.cc:140] P ca7699ae05144c29b50003939e975810: Received BeginTabletCopySession request for tablet a3ce20279e6b41d5a527ba472422314f from peer 1ae50d36df0a4738a6e49318c46a80aa ({username='slave'} at 127.0.99.131:56027)
I20240503 13:59:52.932224 13294 tablet_copy_service.cc:161] P ca7699ae05144c29b50003939e975810: Beginning new tablet copy session on tablet a3ce20279e6b41d5a527ba472422314f from peer 1ae50d36df0a4738a6e49318c46a80aa at {username='slave'} at 127.0.99.131:56027: session id = 1ae50d36df0a4738a6e49318c46a80aa-a3ce20279e6b41d5a527ba472422314f
I20240503 13:59:52.932991 13294 tablet_copy_source_session.cc:205] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:52.933923 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce20279e6b41d5a527ba472422314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:52.935889 14201 tablet_copy_client.cc:783] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:52.936038 14201 tablet_copy_client.cc:647] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:52.937407 14201 tablet_copy_client.cc:523] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:52.938663 14201 tablet_bootstrap.cc:492] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap starting.
I20240503 13:59:52.941871 14201 tablet_bootstrap.cc:492] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:52.942121 14201 tablet_bootstrap.cc:492] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: Bootstrap complete.
I20240503 13:59:52.942281 14201 ts_tablet_manager.cc:1367] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 13:59:52.942692 14201 raft_consensus.cc:348] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } }
I20240503 13:59:52.942793 14201 raft_consensus.cc:729] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Initialized, Role: LEARNER
I20240503 13:59:52.942888 14201 consensus_queue.cc:260] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: true } }
I20240503 13:59:52.943132 14201 ts_tablet_manager.cc:1398] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:52.943776 13294 tablet_copy_service.cc:342] P ca7699ae05144c29b50003939e975810: Request end of tablet copy session 1ae50d36df0a4738a6e49318c46a80aa-a3ce20279e6b41d5a527ba472422314f received from {username='slave'} at 127.0.99.131:56027
I20240503 13:59:52.943868 13294 tablet_copy_service.cc:433] P ca7699ae05144c29b50003939e975810: ending tablet copy session 1ae50d36df0a4738a6e49318c46a80aa-a3ce20279e6b41d5a527ba472422314f on tablet a3ce20279e6b41d5a527ba472422314f with peer 1ae50d36df0a4738a6e49318c46a80aa
I20240503 13:59:53.024916 14194 ts_tablet_manager.cc:902] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Initiating tablet copy from peer 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165)
I20240503 13:59:53.025416 14194 tablet_copy_client.cc:308] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33165
I20240503 13:59:53.026082 13163 tablet_copy_service.cc:140] P 5008569cd3fd47ac8c2f3d2879e3d3f1: Received BeginTabletCopySession request for tablet 46aad7ddf91b4098b9d4ec0c862f3173 from peer 91f8d9c045af4db4a6a3d4d39e5a55e8 ({username='slave'} at 127.0.99.132:57749)
I20240503 13:59:53.026216 13163 tablet_copy_service.cc:161] P 5008569cd3fd47ac8c2f3d2879e3d3f1: Beginning new tablet copy session on tablet 46aad7ddf91b4098b9d4ec0c862f3173 from peer 91f8d9c045af4db4a6a3d4d39e5a55e8 at {username='slave'} at 127.0.99.132:57749: session id = 91f8d9c045af4db4a6a3d4d39e5a55e8-46aad7ddf91b4098b9d4ec0c862f3173
I20240503 13:59:53.027063 13163 tablet_copy_source_session.cc:205] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:53.028215 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46aad7ddf91b4098b9d4ec0c862f3173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:53.030284 14194 tablet_copy_client.cc:783] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:53.030444 14194 tablet_copy_client.cc:647] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:53.031616 14194 tablet_copy_client.cc:523] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:53.032687 14194 tablet_bootstrap.cc:492] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap starting.
I20240503 13:59:53.035777 14194 tablet_bootstrap.cc:492] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:53.036011 14194 tablet_bootstrap.cc:492] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Bootstrap complete.
I20240503 13:59:53.036166 14194 ts_tablet_manager.cc:1367] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 13:59:53.036579 14194 raft_consensus.cc:348] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:53.036684 14194 raft_consensus.cc:729] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91f8d9c045af4db4a6a3d4d39e5a55e8, State: Initialized, Role: LEARNER
I20240503 13:59:53.036773 14194 consensus_queue.cc:260] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: true } }
I20240503 13:59:53.037031 14194 ts_tablet_manager.cc:1398] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:53.037693 13163 tablet_copy_service.cc:342] P 5008569cd3fd47ac8c2f3d2879e3d3f1: Request end of tablet copy session 91f8d9c045af4db4a6a3d4d39e5a55e8-46aad7ddf91b4098b9d4ec0c862f3173 received from {username='slave'} at 127.0.99.132:57749
I20240503 13:59:53.037775 13163 tablet_copy_service.cc:433] P 5008569cd3fd47ac8c2f3d2879e3d3f1: ending tablet copy session 91f8d9c045af4db4a6a3d4d39e5a55e8-46aad7ddf91b4098b9d4ec0c862f3173 on tablet 46aad7ddf91b4098b9d4ec0c862f3173 with peer 91f8d9c045af4db4a6a3d4d39e5a55e8
I20240503 13:59:53.283131 13803 raft_consensus.cc:1203] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.419441 13940 raft_consensus.cc:1203] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 13:59:53.432719 13940 raft_consensus.cc:1203] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.442487 13803 raft_consensus.cc:1203] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.567231 13940 raft_consensus.cc:1203] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:59:53.759797 14272 raft_consensus.cc:1050] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: attempting to promote NON_VOTER 1ae50d36df0a4738a6e49318c46a80aa to VOTER
I20240503 13:59:53.760349 14272 consensus_queue.cc:237] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:53.761734 13803 raft_consensus.cc:1261] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Refusing update from remote peer ca7699ae05144c29b50003939e975810: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:53.762301 14115 consensus_queue.cc:1035] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:53.764473 14272 raft_consensus.cc:2932] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:53.765058 13803 raft_consensus.cc:2932] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:53.767222 13000 catalog_manager.cc:5484] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 reported cstate change: config changed from index 2 to 3, 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:53.836169 14272 raft_consensus.cc:1050] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: attempting to promote NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 to VOTER
I20240503 13:59:53.836547 14272 consensus_queue.cc:237] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:53.837894 13940 raft_consensus.cc:1261] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Refusing update from remote peer ca7699ae05144c29b50003939e975810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:53.838354 14272 consensus_queue.cc:1035] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:53.841228 14271 raft_consensus.cc:2932] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:53.841961 13940 raft_consensus.cc:2932] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:53.844137 12999 catalog_manager.cc:5484] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 reported cstate change: config changed from index 3 to 4, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:53.867836 14299 raft_consensus.cc:1050] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: attempting to promote NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 to VOTER
I20240503 13:59:53.868408 14299 consensus_queue.cc:237] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:53.869694 13940 raft_consensus.cc:1261] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Refusing update from remote peer 5008569cd3fd47ac8c2f3d2879e3d3f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:53.870186 14278 consensus_queue.cc:1035] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:53.872058 14299 raft_consensus.cc:2932] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:53.872550 13940 raft_consensus.cc:2932] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:53.874783 13000 catalog_manager.cc:5484] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: config changed from index 2 to 3, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:53.903024 13274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f54dc5537468409c9fe2117431d58634"
dest_uuid: "ca7699ae05144c29b50003939e975810"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39824
I20240503 13:59:53.903214 13274 raft_consensus.cc:595] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:53.914690 13274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "661cf4663eb04784bd3336c626397d00"
dest_uuid: "ca7699ae05144c29b50003939e975810"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39842
I20240503 13:59:53.914868 13274 raft_consensus.cc:595] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:53.932785 13143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60986ef8847e491ca968d60a5c8d302e"
dest_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49242
I20240503 13:59:53.932957 13143 raft_consensus.cc:595] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:53.952917 14271 raft_consensus.cc:1050] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: attempting to promote NON_VOTER 1ae50d36df0a4738a6e49318c46a80aa to VOTER
I20240503 13:59:53.953253 14271 consensus_queue.cc:237] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:53.954382 13803 raft_consensus.cc:1261] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 LEARNER]: Refusing update from remote peer ca7699ae05144c29b50003939e975810: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:53.954766 14271 consensus_queue.cc:1035] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:53.956497 14272 raft_consensus.cc:2932] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:53.956955 13803 raft_consensus.cc:2932] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:53.959270 13000 catalog_manager.cc:5484] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: config changed from index 2 to 3, 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca7699ae05144c29b50003939e975810" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:54.016219 14124 raft_consensus.cc:1050] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: attempting to promote NON_VOTER 91f8d9c045af4db4a6a3d4d39e5a55e8 to VOTER
I20240503 13:59:54.016606 14124 consensus_queue.cc:237] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.017791 13940 raft_consensus.cc:1261] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 LEARNER]: Refusing update from remote peer 5008569cd3fd47ac8c2f3d2879e3d3f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:59:54.018187 14124 consensus_queue.cc:1035] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:54.020038 14299 raft_consensus.cc:2932] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:54.020501 13940 raft_consensus.cc:2932] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:54.022330 12999 catalog_manager.cc:5484] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 reported cstate change: config changed from index 2 to 3, 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.151494 13274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f54dc5537468409c9fe2117431d58634"
dest_uuid: "ca7699ae05144c29b50003939e975810"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39866
I20240503 13:59:54.151650 13274 raft_consensus.cc:595] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.163007 13274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "661cf4663eb04784bd3336c626397d00"
dest_uuid: "ca7699ae05144c29b50003939e975810"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39884
I20240503 13:59:54.163161 13274 raft_consensus.cc:595] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.174722 13143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46aad7ddf91b4098b9d4ec0c862f3173"
dest_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49262
I20240503 13:59:54.174877 13143 raft_consensus.cc:595] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.186368 13143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60986ef8847e491ca968d60a5c8d302e"
dest_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49272
I20240503 13:59:54.186518 13143 raft_consensus.cc:595] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.192498 14272 raft_consensus.cc:979] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: : Instructing follower 1ae50d36df0a4738a6e49318c46a80aa to start an election
I20240503 13:59:54.192605 14272 raft_consensus.cc:1067] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Signalling peer 1ae50d36df0a4738a6e49318c46a80aa to start an election
I20240503 13:59:54.193219 13803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "661cf4663eb04784bd3336c626397d00"
dest_uuid: "1ae50d36df0a4738a6e49318c46a80aa"
from {username='slave'} at 127.0.99.130:40801
I20240503 13:59:54.193406 13803 raft_consensus.cc:482] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:54.193495 13803 raft_consensus.cc:3037] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.195734 13803 raft_consensus.cc:504] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.196602 13803 leader_election.cc:288] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 2 election: Requested vote from peers ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:54.197119 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661cf4663eb04784bd3336c626397d00" candidate_uuid: "1ae50d36df0a4738a6e49318c46a80aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ca7699ae05144c29b50003939e975810"
I20240503 13:59:54.197315 13274 raft_consensus.cc:3032] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:54.197357 13274 raft_consensus.cc:729] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:54.197515 13274 consensus_queue.cc:260] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.197670 13274 raft_consensus.cc:3037] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.198518 13273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ce20279e6b41d5a527ba472422314f"
dest_uuid: "ca7699ae05144c29b50003939e975810"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39898
I20240503 13:59:54.198633 13273 raft_consensus.cc:595] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.198715 13274 raft_consensus.cc:2454] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae50d36df0a4738a6e49318c46a80aa in term 2.
I20240503 13:59:54.199105 13737 leader_election.cc:302] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1ae50d36df0a4738a6e49318c46a80aa, ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:54.199584 14210 raft_consensus.cc:2781] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:54.199769 14210 raft_consensus.cc:686] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 LEADER]: Becoming Leader. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Running, Role: LEADER
I20240503 13:59:54.199929 14210 consensus_queue.cc:237] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.201761 13000 catalog_manager.cc:5484] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: term changed from 1 to 2, leader changed from ca7699ae05144c29b50003939e975810 (127.0.99.130) to 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131). New cstate: current_term: 2 leader_uuid: "1ae50d36df0a4738a6e49318c46a80aa" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:54.208027 12987 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 661cf4663eb04784bd3336c626397d00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131:41827) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 13:59:54.310484 14278 raft_consensus.cc:979] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: : Instructing follower 91f8d9c045af4db4a6a3d4d39e5a55e8 to start an election
I20240503 13:59:54.310629 14278 raft_consensus.cc:1067] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Signalling peer 91f8d9c045af4db4a6a3d4d39e5a55e8 to start an election
I20240503 13:59:54.311254 13940 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60986ef8847e491ca968d60a5c8d302e"
dest_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
from {username='slave'} at 127.0.99.129:40073
I20240503 13:59:54.311489 13940 raft_consensus.cc:482] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:54.311586 13940 raft_consensus.cc:3037] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.312434 13940 raft_consensus.cc:504] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.313302 13940 leader_election.cc:288] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [CANDIDATE]: Term 2 election: Requested vote from peers 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165)
I20240503 13:59:54.313750 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60986ef8847e491ca968d60a5c8d302e" candidate_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
I20240503 13:59:54.313922 13143 raft_consensus.cc:3032] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:54.313966 13143 raft_consensus.cc:729] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:54.314167 13143 consensus_queue.cc:260] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.314330 13143 raft_consensus.cc:3037] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.315181 13143 raft_consensus.cc:2454] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f8d9c045af4db4a6a3d4d39e5a55e8 in term 2.
I20240503 13:59:54.315596 13875 leader_election.cc:302] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1, 91f8d9c045af4db4a6a3d4d39e5a55e8; no voters:
I20240503 13:59:54.315851 14208 raft_consensus.cc:2781] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:54.316033 14208 raft_consensus.cc:686] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 LEADER]: Becoming Leader. State: Replica: 91f8d9c045af4db4a6a3d4d39e5a55e8, State: Running, Role: LEADER
I20240503 13:59:54.316334 14208 consensus_queue.cc:237] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.318461 13000 catalog_manager.cc:5484] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 reported cstate change: term changed from 1 to 2, leader changed from 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129) to 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.334357 14272 raft_consensus.cc:979] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: : Instructing follower 91f8d9c045af4db4a6a3d4d39e5a55e8 to start an election
I20240503 13:59:54.334486 14272 raft_consensus.cc:1067] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Signalling peer 91f8d9c045af4db4a6a3d4d39e5a55e8 to start an election
I20240503 13:59:54.335005 13940 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f54dc5537468409c9fe2117431d58634"
dest_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
from {username='slave'} at 127.0.99.130:42725
I20240503 13:59:54.335192 13940 raft_consensus.cc:482] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:54.335275 13940 raft_consensus.cc:3037] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.336190 13940 raft_consensus.cc:504] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.336623 13940 leader_election.cc:288] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [CANDIDATE]: Term 2 election: Requested vote from peers ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:54.337090 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f54dc5537468409c9fe2117431d58634" candidate_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ca7699ae05144c29b50003939e975810"
I20240503 13:59:54.337218 13274 raft_consensus.cc:3032] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:54.337265 13274 raft_consensus.cc:729] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:54.337432 13274 consensus_queue.cc:260] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.337577 13274 raft_consensus.cc:3037] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.338415 13274 raft_consensus.cc:2454] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f8d9c045af4db4a6a3d4d39e5a55e8 in term 2.
I20240503 13:59:54.338735 13875 leader_election.cc:302] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 91f8d9c045af4db4a6a3d4d39e5a55e8, ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:54.338937 14208 raft_consensus.cc:2781] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:54.339040 14208 raft_consensus.cc:686] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 LEADER]: Becoming Leader. State: Replica: 91f8d9c045af4db4a6a3d4d39e5a55e8, State: Running, Role: LEADER
I20240503 13:59:54.339279 14208 consensus_queue.cc:237] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.340839 13000 catalog_manager.cc:5484] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 reported cstate change: term changed from 1 to 2, leader changed from ca7699ae05144c29b50003939e975810 (127.0.99.130) to 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.424279 13143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46aad7ddf91b4098b9d4ec0c862f3173"
dest_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49288
I20240503 13:59:54.424441 13143 raft_consensus.cc:595] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.441669 13274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ce20279e6b41d5a527ba472422314f"
dest_uuid: "ca7699ae05144c29b50003939e975810"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39922
I20240503 13:59:54.441809 13274 raft_consensus.cc:595] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:54.544481 14299 raft_consensus.cc:979] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: : Instructing follower 91f8d9c045af4db4a6a3d4d39e5a55e8 to start an election
I20240503 13:59:54.544608 14299 raft_consensus.cc:1067] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Signalling peer 91f8d9c045af4db4a6a3d4d39e5a55e8 to start an election
I20240503 13:59:54.545075 13940 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46aad7ddf91b4098b9d4ec0c862f3173"
dest_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8"
from {username='slave'} at 127.0.99.129:40073
I20240503 13:59:54.545238 13940 raft_consensus.cc:482] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:54.545300 13940 raft_consensus.cc:3037] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.546268 13940 raft_consensus.cc:504] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.546564 13940 leader_election.cc:288] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [CANDIDATE]: Term 2 election: Requested vote from peers 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165)
I20240503 13:59:54.546923 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46aad7ddf91b4098b9d4ec0c862f3173" candidate_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1"
I20240503 13:59:54.547044 13143 raft_consensus.cc:3032] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:54.547091 13143 raft_consensus.cc:729] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5008569cd3fd47ac8c2f3d2879e3d3f1, State: Running, Role: LEADER
I20240503 13:59:54.547245 13143 consensus_queue.cc:260] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.547384 13143 raft_consensus.cc:3037] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.548254 13143 raft_consensus.cc:2454] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f8d9c045af4db4a6a3d4d39e5a55e8 in term 2.
I20240503 13:59:54.548553 13875 leader_election.cc:302] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5008569cd3fd47ac8c2f3d2879e3d3f1, 91f8d9c045af4db4a6a3d4d39e5a55e8; no voters:
I20240503 13:59:54.548732 14208 raft_consensus.cc:2781] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:54.548828 14208 raft_consensus.cc:686] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 LEADER]: Becoming Leader. State: Replica: 91f8d9c045af4db4a6a3d4d39e5a55e8, State: Running, Role: LEADER
I20240503 13:59:54.548988 14208 consensus_queue.cc:237] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.550397 13000 catalog_manager.cc:5484] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 reported cstate change: term changed from 1 to 2, leader changed from 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129) to 91f8d9c045af4db4a6a3d4d39e5a55e8 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.562829 14115 raft_consensus.cc:979] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: : Instructing follower 1ae50d36df0a4738a6e49318c46a80aa to start an election
I20240503 13:59:54.562950 14115 raft_consensus.cc:1067] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Signalling peer 1ae50d36df0a4738a6e49318c46a80aa to start an election
I20240503 13:59:54.563468 13803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ce20279e6b41d5a527ba472422314f"
dest_uuid: "1ae50d36df0a4738a6e49318c46a80aa"
from {username='slave'} at 127.0.99.130:40801
I20240503 13:59:54.563627 13803 raft_consensus.cc:482] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:54.563695 13803 raft_consensus.cc:3037] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.564633 13803 raft_consensus.cc:504] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.565106 13803 leader_election.cc:288] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 2 election: Requested vote from peers ca7699ae05144c29b50003939e975810 (127.0.99.130:45279)
I20240503 13:59:54.565515 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce20279e6b41d5a527ba472422314f" candidate_uuid: "1ae50d36df0a4738a6e49318c46a80aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ca7699ae05144c29b50003939e975810"
I20240503 13:59:54.565667 13274 raft_consensus.cc:3032] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:54.565729 13274 raft_consensus.cc:729] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca7699ae05144c29b50003939e975810, State: Running, Role: LEADER
I20240503 13:59:54.565886 13274 consensus_queue.cc:260] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.566084 13274 raft_consensus.cc:3037] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:54.566954 13274 raft_consensus.cc:2454] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae50d36df0a4738a6e49318c46a80aa in term 2.
I20240503 13:59:54.567262 13737 leader_election.cc:302] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1ae50d36df0a4738a6e49318c46a80aa, ca7699ae05144c29b50003939e975810; no voters:
I20240503 13:59:54.567442 14210 raft_consensus.cc:2781] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:54.567528 14210 raft_consensus.cc:686] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 2 LEADER]: Becoming Leader. State: Replica: 1ae50d36df0a4738a6e49318c46a80aa, State: Running, Role: LEADER
I20240503 13:59:54.567674 14210 consensus_queue.cc:237] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.569190 13000 catalog_manager.cc:5484] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: term changed from 1 to 2, leader changed from ca7699ae05144c29b50003939e975810 (127.0.99.130) to 1ae50d36df0a4738a6e49318c46a80aa (127.0.99.131). New cstate: current_term: 2 leader_uuid: "1ae50d36df0a4738a6e49318c46a80aa" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.728885 13274 raft_consensus.cc:1261] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Refusing update from remote peer 91f8d9c045af4db4a6a3d4d39e5a55e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:59:54.729647 14208 consensus_queue.cc:1035] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:54.778671 13274 raft_consensus.cc:1261] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Refusing update from remote peer 1ae50d36df0a4738a6e49318c46a80aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:54.779246 14210 consensus_queue.cc:1035] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.785193 13143 raft_consensus.cc:1261] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Refusing update from remote peer 91f8d9c045af4db4a6a3d4d39e5a55e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:54.785775 14432 consensus_queue.cc:1035] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.878705 13803 consensus_queue.cc:237] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:54.879377 14210 raft_consensus.cc:2932] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ca7699ae05144c29b50003939e975810 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:54.880990 12987 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 661cf4663eb04784bd3336c626397d00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:54.881944 13000 catalog_manager.cc:5484] T 661cf4663eb04784bd3336c626397d00 P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: config changed from index 3 to 5, VOTER ca7699ae05144c29b50003939e975810 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "1ae50d36df0a4738a6e49318c46a80aa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.885092 13253 tablet_service.cc:1514] Processing DeleteTablet for tablet 661cf4663eb04784bd3336c626397d00 with delete_type TABLET_DATA_TOMBSTONED (TS ca7699ae05144c29b50003939e975810 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39618
I20240503 13:59:54.885674 14509 tablet_replica.cc:330] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: stopping tablet replica
I20240503 13:59:54.886058 14509 raft_consensus.cc:2229] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.886279 14509 raft_consensus.cc:2258] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.887404 14509 ts_tablet_manager.cc:1875] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.890383 14509 ts_tablet_manager.cc:1888] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:54.890496 14509 log.cc:1196] T 661cf4663eb04784bd3336c626397d00 P ca7699ae05144c29b50003939e975810: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/661cf4663eb04784bd3336c626397d00
I20240503 13:59:54.891173 12987 catalog_manager.cc:4830] TS ca7699ae05144c29b50003939e975810 (127.0.99.130:45279): tablet 661cf4663eb04784bd3336c626397d00 (table rebalance_test_table_2 [id=18d432015e4149348a15b645e6b8697c]) successfully deleted
I20240503 13:59:54.967511 13940 consensus_queue.cc:237] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.968240 14432 raft_consensus.cc:2932] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:54.969130 13143 raft_consensus.cc:1261] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Refusing update from remote peer 91f8d9c045af4db4a6a3d4d39e5a55e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:54.969637 14432 consensus_queue.cc:1035] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5008569cd3fd47ac8c2f3d2879e3d3f1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:54.969897 12985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60986ef8847e491ca968d60a5c8d302e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:54.970947 13000 catalog_manager.cc:5484] T 60986ef8847e491ca968d60a5c8d302e P 91f8d9c045af4db4a6a3d4d39e5a55e8 reported cstate change: config changed from index 3 to 5, VOTER 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:54.973591 13123 tablet_service.cc:1514] Processing DeleteTablet for tablet 60986ef8847e491ca968d60a5c8d302e with delete_type TABLET_DATA_TOMBSTONED (TS 5008569cd3fd47ac8c2f3d2879e3d3f1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49016
I20240503 13:59:54.974210 14546 tablet_replica.cc:330] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: stopping tablet replica
I20240503 13:59:54.974586 14546 raft_consensus.cc:2229] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:54.974779 14546 raft_consensus.cc:2258] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:54.975834 14546 ts_tablet_manager.cc:1875] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:54.978163 14546 ts_tablet_manager.cc:1888] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:54.978289 14546 log.cc:1196] T 60986ef8847e491ca968d60a5c8d302e P 5008569cd3fd47ac8c2f3d2879e3d3f1: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/60986ef8847e491ca968d60a5c8d302e
I20240503 13:59:54.979200 12987 catalog_manager.cc:4830] TS 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165): tablet 60986ef8847e491ca968d60a5c8d302e (table rebalance_test_table_2 [id=18d432015e4149348a15b645e6b8697c]) successfully deleted
I20240503 13:59:54.995699 13940 consensus_queue.cc:237] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:54.996446 14208 raft_consensus.cc:2932] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ca7699ae05144c29b50003939e975810 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:54.997900 12985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f54dc5537468409c9fe2117431d58634 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:54.998822 12999 catalog_manager.cc:5484] T f54dc5537468409c9fe2117431d58634 P 91f8d9c045af4db4a6a3d4d39e5a55e8 reported cstate change: config changed from index 4 to 6, VOTER ca7699ae05144c29b50003939e975810 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:55.001192 13253 tablet_service.cc:1514] Processing DeleteTablet for tablet f54dc5537468409c9fe2117431d58634 with delete_type TABLET_DATA_TOMBSTONED (TS ca7699ae05144c29b50003939e975810 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39618
I20240503 13:59:55.001436 14509 tablet_replica.cc:330] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: stopping tablet replica
I20240503 13:59:55.001574 14509 raft_consensus.cc:2229] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.001683 14509 raft_consensus.cc:2258] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.002295 14509 ts_tablet_manager.cc:1875] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.004299 14509 ts_tablet_manager.cc:1888] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:59:55.004382 14509 log.cc:1196] T f54dc5537468409c9fe2117431d58634 P ca7699ae05144c29b50003939e975810: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f54dc5537468409c9fe2117431d58634
I20240503 13:59:55.004958 12987 catalog_manager.cc:4830] TS ca7699ae05144c29b50003939e975810 (127.0.99.130:45279): tablet f54dc5537468409c9fe2117431d58634 (table rebalance_test_table_1 [id=76fbe485b57a437cab200393da94cf52]) successfully deleted
I20240503 13:59:55.129900 13274 raft_consensus.cc:1261] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Refusing update from remote peer 1ae50d36df0a4738a6e49318c46a80aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:59:55.130514 14429 consensus_queue.cc:1035] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7699ae05144c29b50003939e975810" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45279 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:55.162441 13940 consensus_queue.cc:237] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } }
I20240503 13:59:55.163097 14545 raft_consensus.cc:2932] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } } }
I20240503 13:59:55.164695 12985 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46aad7ddf91b4098b9d4ec0c862f3173 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:55.165684 13001 catalog_manager.cc:5484] T 46aad7ddf91b4098b9d4ec0c862f3173 P 91f8d9c045af4db4a6a3d4d39e5a55e8 reported cstate change: config changed from index 3 to 5, VOTER 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "91f8d9c045af4db4a6a3d4d39e5a55e8" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:55.168851 13123 tablet_service.cc:1514] Processing DeleteTablet for tablet 46aad7ddf91b4098b9d4ec0c862f3173 with delete_type TABLET_DATA_TOMBSTONED (TS 5008569cd3fd47ac8c2f3d2879e3d3f1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49016
I20240503 13:59:55.169081 14546 tablet_replica.cc:330] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: stopping tablet replica
I20240503 13:59:55.169193 14546 raft_consensus.cc:2229] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.169289 14546 raft_consensus.cc:2258] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.169900 14546 ts_tablet_manager.cc:1875] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.171758 14546 ts_tablet_manager.cc:1888] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:55.171837 14546 log.cc:1196] T 46aad7ddf91b4098b9d4ec0c862f3173 P 5008569cd3fd47ac8c2f3d2879e3d3f1: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/46aad7ddf91b4098b9d4ec0c862f3173
I20240503 13:59:55.172470 12987 catalog_manager.cc:4830] TS 5008569cd3fd47ac8c2f3d2879e3d3f1 (127.0.99.129:33165): tablet 46aad7ddf91b4098b9d4ec0c862f3173 (table rebalance_test_table_0 [id=914a6956449240e1afba80f78f23c7a8]) successfully deleted
I20240503 13:59:55.209233 13803 consensus_queue.cc:237] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } }
I20240503 13:59:55.210050 14548 raft_consensus.cc:2932] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ca7699ae05144c29b50003939e975810 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } } }
I20240503 13:59:55.211627 12987 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3ce20279e6b41d5a527ba472422314f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 13:59:55.212618 13001 catalog_manager.cc:5484] T a3ce20279e6b41d5a527ba472422314f P 1ae50d36df0a4738a6e49318c46a80aa reported cstate change: config changed from index 3 to 5, VOTER ca7699ae05144c29b50003939e975810 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "1ae50d36df0a4738a6e49318c46a80aa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1ae50d36df0a4738a6e49318c46a80aa" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:55.215238 13253 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ce20279e6b41d5a527ba472422314f with delete_type TABLET_DATA_TOMBSTONED (TS ca7699ae05144c29b50003939e975810 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39618
I20240503 13:59:55.215464 14509 tablet_replica.cc:330] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: stopping tablet replica
I20240503 13:59:55.215595 14509 raft_consensus.cc:2229] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.215708 14509 raft_consensus.cc:2258] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.216226 14509 ts_tablet_manager.cc:1875] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:55.218076 14509 ts_tablet_manager.cc:1888] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 13:59:55.218158 14509 log.cc:1196] T a3ce20279e6b41d5a527ba472422314f P ca7699ae05144c29b50003939e975810: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3ce20279e6b41d5a527ba472422314f
I20240503 13:59:55.218713 12987 catalog_manager.cc:4830] TS ca7699ae05144c29b50003939e975810 (127.0.99.130:45279): tablet a3ce20279e6b41d5a527ba472422314f (table rebalance_test_table_3 [id=448a9a630f3e460a824c802b0b1ff382]) successfully deleted
I20240503 13:59:55.442111 13783 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.444520 13123 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:55.446151 13920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.446952 13253 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.530606 13783 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.530678 13920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.533413 13253 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.543066 13123 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:55.632117 13920 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.632745 13783 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.634404 13123 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:55.636574 13253 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
cd0fcedd694548f8bcce64315ec09552 | 127.0.99.190:42483 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35095 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1ae50d36df0a4738a6e49318c46a80aa | 127.0.99.131:41827 | HEALTHY | <none> | 3 | 0
5008569cd3fd47ac8c2f3d2879e3d3f1 | 127.0.99.129:33165 | HEALTHY | <none> | 4 | 0
91f8d9c045af4db4a6a3d4d39e5a55e8 | 127.0.99.132:34417 | HEALTHY | <none> | 3 | 0
ca7699ae05144c29b50003939e975810 | 127.0.99.130:45279 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:33165
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:45279
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:41827
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:34417
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:33165
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:45279
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:41827
server_dump_info_path | /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:34417
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35095 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:59:55.663385 398 log_verifier.cc:126] Checking tablet 1c43637c1afe4ec5b01e5dda46ed9086
I20240503 13:59:55.671664 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 1c43637c1afe4ec5b01e5dda46ed9086
I20240503 13:59:55.671741 398 log_verifier.cc:126] Checking tablet 46aad7ddf91b4098b9d4ec0c862f3173
I20240503 13:59:55.678990 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 46aad7ddf91b4098b9d4ec0c862f3173
I20240503 13:59:55.679068 398 log_verifier.cc:126] Checking tablet 4e8af25de83048a48ba662dbbd6b551e
I20240503 13:59:55.686116 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4e8af25de83048a48ba662dbbd6b551e
I20240503 13:59:55.686187 398 log_verifier.cc:126] Checking tablet 558c698a762b4742971f25f5d997013c
I20240503 13:59:55.693289 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 558c698a762b4742971f25f5d997013c
I20240503 13:59:55.693365 398 log_verifier.cc:126] Checking tablet 60986ef8847e491ca968d60a5c8d302e
I20240503 13:59:55.700489 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 60986ef8847e491ca968d60a5c8d302e
I20240503 13:59:55.700574 398 log_verifier.cc:126] Checking tablet 636cea461e514757a78ea545202d5148
I20240503 13:59:55.707544 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 636cea461e514757a78ea545202d5148
I20240503 13:59:55.707615 398 log_verifier.cc:126] Checking tablet 661cf4663eb04784bd3336c626397d00
I20240503 13:59:55.714988 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 661cf4663eb04784bd3336c626397d00
I20240503 13:59:55.715055 398 log_verifier.cc:126] Checking tablet 8bd37b00b3ee49b1bdda91019af39c57
I20240503 13:59:55.722505 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8bd37b00b3ee49b1bdda91019af39c57
I20240503 13:59:55.722585 398 log_verifier.cc:126] Checking tablet 94180255c91e4ff49d20f0fef0c71c4b
I20240503 13:59:55.730609 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 94180255c91e4ff49d20f0fef0c71c4b
I20240503 13:59:55.730679 398 log_verifier.cc:126] Checking tablet a3ce20279e6b41d5a527ba472422314f
I20240503 13:59:55.737960 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet a3ce20279e6b41d5a527ba472422314f
I20240503 13:59:55.738062 398 log_verifier.cc:126] Checking tablet e6db7b0e734c4165abe1a4b3acc1511e
I20240503 13:59:55.744922 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e6db7b0e734c4165abe1a4b3acc1511e
I20240503 13:59:55.744995 398 log_verifier.cc:126] Checking tablet f4844a0937c7491e955ee51beceada38
I20240503 13:59:55.752080 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet f4844a0937c7491e955ee51beceada38
I20240503 13:59:55.752153 398 log_verifier.cc:126] Checking tablet f54dc5537468409c9fe2117431d58634
I20240503 13:59:55.759347 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f54dc5537468409c9fe2117431d58634
I20240503 13:59:55.759945 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 13061
I20240503 13:59:55.765619 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 13192
I20240503 13:59:55.774456 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 13720
I20240503 13:59:55.780069 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 13853
I20240503 13:59:55.785461 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 12970
2024-05-03T13:59:55Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (9892 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (9892 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240503 13:59:55.815812 398 test_util.cc:268] Using random seed: 577724286
I20240503 13:59:55.816618 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:55.816661 398 ts_itest-base.cc:113] --------------
I20240503 13:59:55.816671 398 ts_itest-base.cc:114] 6 tablet servers
I20240503 13:59:55.816680 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:59:55.816689 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:55Z Disabled control of system clock
I20240503 13:59:55.824980 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37509
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37509
--location_mapping_cmd=/tmp/dist-test-taskmJWZN5/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 13:59:55.938210 14683 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 13:59:55.938515 14683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:55.938557 14683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:55.938616 14683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:55.941830 14683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:55.941933 14683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:55.942003 14683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:55.942037 14683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:55.942075 14683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:55.946056 14683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskmJWZN5/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37509
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:55.947065 14683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:55.948299 14683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:55.956508 14688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.956928 14689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.957872 14691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.958586 14683 server_base.cc:1034] running on GCE node
I20240503 13:59:55.959193 14683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:55.960237 14683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:55.961452 14683 hybrid_clock.cc:648] HybridClock initialized: now 1714744795961402 us; error 48 us; skew 500 ppm
I20240503 13:59:55.967434 14683 webserver.cc:460] Webserver started at http://127.0.99.190:38109/ using document root <none> and password file <none>
I20240503 13:59:55.968094 14683 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.968161 14683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.968340 14683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.970297 14683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b199ccafb2894f4fb6091c6a5dad6361"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6fwv"
I20240503 13:59:55.970827 14683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b199ccafb2894f4fb6091c6a5dad6361"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6fwv"
I20240503 13:59:55.974545 14683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 13:59:55.977061 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.978371 14683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:55.978531 14683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b199ccafb2894f4fb6091c6a5dad6361"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6fwv"
I20240503 13:59:55.978646 14683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.998338 14683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.999090 14683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:55.999266 14683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:56.005903 14683 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37509
I20240503 13:59:56.005924 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37509 every 8 connection(s)
I20240503 13:59:56.007057 14683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:56.009958 14750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:56.010224 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 14683
I20240503 13:59:56.010371 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:56.017436 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361: Bootstrap starting.
I20240503 13:59:56.020236 14750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:56.021170 14750 log.cc:824] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:56.023108 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361: No bootstrap required, opened a new log
I20240503 13:59:56.025882 14750 raft_consensus.cc:348] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b199ccafb2894f4fb6091c6a5dad6361" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37509 } }
I20240503 13:59:56.026157 14750 raft_consensus.cc:374] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:56.026212 14750 raft_consensus.cc:729] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b199ccafb2894f4fb6091c6a5dad6361, State: Initialized, Role: FOLLOWER
I20240503 13:59:56.026666 14750 consensus_queue.cc:260] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b199ccafb2894f4fb6091c6a5dad6361" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37509 } }
I20240503 13:59:56.026813 14750 raft_consensus.cc:388] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:56.026885 14750 raft_consensus.cc:482] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:56.027009 14750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:56.027930 14750 raft_consensus.cc:504] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b199ccafb2894f4fb6091c6a5dad6361" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37509 } }
I20240503 13:59:56.028329 14750 leader_election.cc:302] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b199ccafb2894f4fb6091c6a5dad6361; no voters:
I20240503 13:59:56.028641 14750 leader_election.cc:288] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:56.029065 14755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:56.029261 14755 raft_consensus.cc:686] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [term 1 LEADER]: Becoming Leader. State: Replica: b199ccafb2894f4fb6091c6a5dad6361, State: Running, Role: LEADER
I20240503 13:59:56.029594 14755 consensus_queue.cc:237] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b199ccafb2894f4fb6091c6a5dad6361" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37509 } }
I20240503 13:59:56.030028 14750 sys_catalog.cc:564] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:56.031404 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b199ccafb2894f4fb6091c6a5dad6361. Latest consensus state: current_term: 1 leader_uuid: "b199ccafb2894f4fb6091c6a5dad6361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b199ccafb2894f4fb6091c6a5dad6361" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37509 } } }
I20240503 13:59:56.031407 14756 sys_catalog.cc:455] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b199ccafb2894f4fb6091c6a5dad6361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b199ccafb2894f4fb6091c6a5dad6361" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37509 } } }
I20240503 13:59:56.031575 14756 sys_catalog.cc:458] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:56.031575 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:56.032024 14764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:56.035297 14764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:56.041069 14764 catalog_manager.cc:1347] Generated new cluster ID: 327a6c125cd640fdb9a9408940ce9c43
I20240503 13:59:56.041152 14764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:56.050710 14764 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:56.051709 14764 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:56.059084 14764 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b199ccafb2894f4fb6091c6a5dad6361: Generated new TSK 0
I20240503 13:59:56.060140 14764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:56.069726 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 13:59:56.182591 14774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:56.182870 14774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:56.182926 14774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:56.186345 14774 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:56.186434 14774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:56.186514 14774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:56.190547 14774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.14774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:56.191638 14774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:56.192864 14774 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:56.202199 14782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.202587 14779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.203473 14780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:56.204149 14774 server_base.cc:1034] running on GCE node
I20240503 13:59:56.204627 14774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:56.205296 14774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:56.206449 14774 hybrid_clock.cc:648] HybridClock initialized: now 1714744796206421 us; error 45 us; skew 500 ppm
I20240503 13:59:56.208623 14774 webserver.cc:460] Webserver started at http://127.0.99.129:37781/ using document root <none> and password file <none>
I20240503 13:59:56.209216 14774 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:56.209276 14774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:56.209484 14774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:56.211318 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6ecce61b6a854f02a65cfabb70a738ea"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.211848 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6ecce61b6a854f02a65cfabb70a738ea"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.215811 14774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 13:59:56.218448 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.219698 14774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:56.219822 14774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6ecce61b6a854f02a65cfabb70a738ea"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.219936 14774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:56.252777 14774 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:56.253540 14774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:56.253723 14774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:56.254384 14774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:56.255414 14774 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:56.255467 14774 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.255513 14774 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:56.255542 14774 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.264079 14774 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43449
I20240503 13:59:56.264111 14901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43449 every 8 connection(s)
I20240503 13:59:56.265170 14774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:56.266114 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 14774
I20240503 13:59:56.266223 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:56.268811 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:56.276192 14902 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 13:59:56.276497 14902 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:56.277151 14902 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 13:59:56.315938 14714 ts_manager.cc:194] Registered new tserver with Master: 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449)
I20240503 13:59:56.317760 14714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47607
W20240503 13:59:56.387117 14905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:56.387388 14905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:56.387434 14905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:56.390619 14905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:56.390704 14905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:56.390780 14905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:56.394613 14905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.14905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:56.395601 14905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:56.396781 14905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:56.406069 14914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.406790 14912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.407445 14911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:56.408293 14905 server_base.cc:1034] running on GCE node
I20240503 13:59:56.408761 14905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:56.409397 14905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:56.410557 14905 hybrid_clock.cc:648] HybridClock initialized: now 1714744796410536 us; error 36 us; skew 500 ppm
I20240503 13:59:56.412608 14905 webserver.cc:460] Webserver started at http://127.0.99.130:34627/ using document root <none> and password file <none>
I20240503 13:59:56.413197 14905 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:56.413261 14905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:56.413460 14905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:56.415280 14905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "89314f1a84d8474dba0db8615c043d1a"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.415774 14905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "89314f1a84d8474dba0db8615c043d1a"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.419327 14905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:56.421692 14920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.422801 14905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:56.422914 14905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "89314f1a84d8474dba0db8615c043d1a"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.423005 14905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:56.450781 14905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:56.451478 14905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:56.451648 14905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:56.452200 14905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:56.453142 14905 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:56.453190 14905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.453243 14905 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:56.453269 14905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.461508 14905 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:44623
I20240503 13:59:56.461552 15033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:44623 every 8 connection(s)
I20240503 13:59:56.462558 14905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:56.465365 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 14905
I20240503 13:59:56.465492 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:56.467732 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:56.472443 15034 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 13:59:56.472687 15034 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:56.473253 15034 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 13:59:56.521296 14714 ts_manager.cc:194] Registered new tserver with Master: 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 13:59:56.522154 14714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:46139
W20240503 13:59:56.592706 15037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:56.592949 15037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:56.592995 15037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:56.596072 15037 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:56.596150 15037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:56.596231 15037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:56.599900 15037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.15037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:56.600860 15037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:56.602066 15037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:56.611117 15043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.611495 15044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.612437 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:56.613078 15037 server_base.cc:1034] running on GCE node
I20240503 13:59:56.613562 15037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:56.614230 15037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:56.615398 15037 hybrid_clock.cc:648] HybridClock initialized: now 1714744796615381 us; error 33 us; skew 500 ppm
I20240503 13:59:56.617491 15037 webserver.cc:460] Webserver started at http://127.0.99.131:43135/ using document root <none> and password file <none>
I20240503 13:59:56.618105 15037 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:56.618166 15037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:56.618333 15037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:56.620030 15037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8614bbdff011473abb0f0cfce354d4a6"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.620491 15037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8614bbdff011473abb0f0cfce354d4a6"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.624039 15037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:56.626310 15052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.627365 15037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:59:56.627506 15037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8614bbdff011473abb0f0cfce354d4a6"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.627614 15037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:56.640193 15037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:56.640949 15037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:56.641111 15037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:56.641681 15037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:56.642652 15037 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:56.642704 15037 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.642748 15037 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:56.642776 15037 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.650904 15037 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45503
I20240503 13:59:56.650928 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45503 every 8 connection(s)
I20240503 13:59:56.651932 15037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:56.654127 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 15037
I20240503 13:59:56.654258 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:56.656549 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:56.664333 15166 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 13:59:56.664589 15166 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:56.665187 15166 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 13:59:56.699386 14714 ts_manager.cc:194] Registered new tserver with Master: 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 13:59:56.700132 14714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:56431
W20240503 13:59:56.773250 15169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:56.773531 15169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:56.773594 15169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:56.776787 15169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:56.776885 15169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:56.776984 15169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:56.780895 15169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.15169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:56.781951 15169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:56.783210 15169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:56.792428 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.793339 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.793939 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:56.794803 15169 server_base.cc:1034] running on GCE node
I20240503 13:59:56.795264 15169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:56.795913 15169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:56.797071 15169 hybrid_clock.cc:648] HybridClock initialized: now 1714744796797053 us; error 35 us; skew 500 ppm
I20240503 13:59:56.799176 15169 webserver.cc:460] Webserver started at http://127.0.99.132:40767/ using document root <none> and password file <none>
I20240503 13:59:56.799841 15169 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:56.799907 15169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:56.800086 15169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:56.801844 15169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "35b5bdde0c49481e8d69060ad4d1d81d"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.802371 15169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "35b5bdde0c49481e8d69060ad4d1d81d"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.805902 15169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:56.808310 15184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.809392 15169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:56.809526 15169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "35b5bdde0c49481e8d69060ad4d1d81d"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:56.809620 15169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:56.825165 15169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:56.825848 15169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:56.826086 15169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:56.826675 15169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:56.827641 15169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:56.827689 15169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.827745 15169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:56.827771 15169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.836445 15169 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:32851
I20240503 13:59:56.836467 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:32851 every 8 connection(s)
I20240503 13:59:56.837517 15169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:56.845268 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 15169
I20240503 13:59:56.845396 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:56.846495 15298 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 13:59:56.846743 15298 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:56.847326 15298 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 13:59:56.847471 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:56.897212 14714 ts_manager.cc:194] Registered new tserver with Master: 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851)
I20240503 13:59:56.898036 14714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:33489
W20240503 13:59:56.975019 15301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:56.975293 15301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:56.975363 15301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:56.978617 15301 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:56.978709 15301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:56.978798 15301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:56.982668 15301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.15301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:56.983654 15301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:56.984827 15301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:56.993695 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.994258 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.994997 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:56.995713 15301 server_base.cc:1034] running on GCE node
I20240503 13:59:56.996170 15301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:56.996798 15301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:56.997946 15301 hybrid_clock.cc:648] HybridClock initialized: now 1714744796997920 us; error 46 us; skew 500 ppm
I20240503 13:59:56.999989 15301 webserver.cc:460] Webserver started at http://127.0.99.133:46035/ using document root <none> and password file <none>
I20240503 13:59:57.000559 15301 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:57.000615 15301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:57.000802 15301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:57.002606 15301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fb0b622a674141069043d816ad306f3e"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:57.003093 15301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fb0b622a674141069043d816ad306f3e"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:57.006850 15301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 13:59:57.009232 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:57.010408 15301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:57.010551 15301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fb0b622a674141069043d816ad306f3e"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 13:59:57.010651 15301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:57.034466 15301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:57.035101 15301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:57.035262 15301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:57.035823 15301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:57.036751 15301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:57.036798 15301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:57.036849 15301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:57.036875 15301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:57.045275 15301 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46269
I20240503 13:59:57.045305 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46269 every 8 connection(s)
I20240503 13:59:57.046301 15301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:57.054212 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 15301
I20240503 13:59:57.054347 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:57.055460 15430 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 13:59:57.055697 15430 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:57.056375 15430 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 13:59:57.056695 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 13:59:57.092669 14714 ts_manager.cc:194] Registered new tserver with Master: fb0b622a674141069043d816ad306f3e (127.0.99.133:46269)
I20240503 13:59:57.093422 14714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:60619
W20240503 13:59:57.170751 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:57.171037 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:57.171103 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:57.174423 15433 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:57.174515 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:57.174609 15433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:57.178592 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.15433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:57.179673 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:57.181010 15433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:57.190572 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:57.191417 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:57.192268 15433 server_base.cc:1034] running on GCE node
W20240503 13:59:57.192833 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:57.193301 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:57.193904 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:57.195075 15433 hybrid_clock.cc:648] HybridClock initialized: now 1714744797195053 us; error 38 us; skew 500 ppm
I20240503 13:59:57.197156 15433 webserver.cc:460] Webserver started at http://127.0.99.134:38757/ using document root <none> and password file <none>
I20240503 13:59:57.197739 15433 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:57.197796 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:57.198024 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:57.199981 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e90a2a2980f84d49a2532f143f71e753"
format_stamp: "Formatted at 2024-05-03 13:59:57 on dist-test-slave-6fwv"
I20240503 13:59:57.200520 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e90a2a2980f84d49a2532f143f71e753"
format_stamp: "Formatted at 2024-05-03 13:59:57 on dist-test-slave-6fwv"
I20240503 13:59:57.204083 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 13:59:57.206342 15448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:57.207418 15433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:57.207547 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e90a2a2980f84d49a2532f143f71e753"
format_stamp: "Formatted at 2024-05-03 13:59:57 on dist-test-slave-6fwv"
I20240503 13:59:57.207640 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:57.221403 15433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:57.222133 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:57.222299 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:57.222862 15433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:57.223870 15433 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:57.223920 15433 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:57.223976 15433 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:57.224002 15433 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:57.232309 15433 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38635
I20240503 13:59:57.232352 15561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38635 every 8 connection(s)
I20240503 13:59:57.233301 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:57.242913 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 15433
I20240503 13:59:57.243026 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:59:57.243162 15562 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 13:59:57.243423 15562 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:57.244122 15562 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 13:59:57.278574 14713 ts_manager.cc:194] Registered new tserver with Master: e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635)
I20240503 13:59:57.279310 14713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:47695
I20240503 13:59:57.282035 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 13:59:57.291482 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 13:59:57.291577 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 15169
I20240503 13:59:57.296753 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 15433
I20240503 13:59:57.301812 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 15301
I20240503 13:59:57.320552 14902 heartbeater.cc:503] Master 127.0.99.190:37509 was elected leader, sending a full tablet report...
I20240503 13:59:57.524587 15034 heartbeater.cc:503] Master 127.0.99.190:37509 was elected leader, sending a full tablet report...
I20240503 13:59:57.702903 15166 heartbeater.cc:503] Master 127.0.99.190:37509 was elected leader, sending a full tablet report...
I20240503 14:00:01.058635 14713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39186:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:00:01.062144 14713 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:01.080600 14968 tablet_service.cc:1467] Processing CreateTablet for tablet 274cf23ceaaa4a049f8c35689918d7b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.080720 14835 tablet_service.cc:1467] Processing CreateTablet for tablet eb5245fe54de4741ada591ab9576e699 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.080710 14836 tablet_service.cc:1467] Processing CreateTablet for tablet 274cf23ceaaa4a049f8c35689918d7b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.081045 15098 tablet_service.cc:1467] Processing CreateTablet for tablet 6d68b8332ce64eb2935deba765c3946a (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.081688 14835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5245fe54de4741ada591ab9576e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.081712 14968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274cf23ceaaa4a049f8c35689918d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.081741 15098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d68b8332ce64eb2935deba765c3946a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.082074 14836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274cf23ceaaa4a049f8c35689918d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.082332 15100 tablet_service.cc:1467] Processing CreateTablet for tablet 274cf23ceaaa4a049f8c35689918d7b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.082581 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274cf23ceaaa4a049f8c35689918d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.082990 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 6d68b8332ce64eb2935deba765c3946a (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.083276 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d68b8332ce64eb2935deba765c3946a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.086328 15099 tablet_service.cc:1467] Processing CreateTablet for tablet eb5245fe54de4741ada591ab9576e699 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.086374 14834 tablet_service.cc:1467] Processing CreateTablet for tablet 6d68b8332ce64eb2935deba765c3946a (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.086602 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5245fe54de4741ada591ab9576e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.086616 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d68b8332ce64eb2935deba765c3946a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.088078 14967 tablet_service.cc:1467] Processing CreateTablet for tablet eb5245fe54de4741ada591ab9576e699 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.088361 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5245fe54de4741ada591ab9576e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.092381 15581 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:01.093416 15582 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:01.094327 15583 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:01.094661 15581 tablet_bootstrap.cc:654] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.095311 15582 tablet_bootstrap.cc:654] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.095538 15581 log.cc:824] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:01.095991 15582 log.cc:824] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:01.096340 15583 tablet_bootstrap.cc:654] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.097167 15583 log.cc:824] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:01.097528 15582 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:01.097527 15581 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:01.097726 15581 ts_tablet_manager.cc:1367] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:00:01.097751 15582 ts_tablet_manager.cc:1367] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:00:01.099328 15583 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:01.099787 15583 ts_tablet_manager.cc:1367] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:00:01.101282 15582 raft_consensus.cc:348] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.101521 15582 raft_consensus.cc:374] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.101614 15582 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.101538 15581 raft_consensus.cc:348] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.101771 15581 raft_consensus.cc:374] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.101874 15581 raft_consensus.cc:729] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.102170 15582 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.102384 15581 consensus_queue.cc:260] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.103310 15581 ts_tablet_manager.cc:1398] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240503 14:00:01.103281 15582 ts_tablet_manager.cc:1398] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:00:01.103652 15581 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:01.103724 15583 raft_consensus.cc:348] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.104019 15583 raft_consensus.cc:374] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.104123 15583 raft_consensus.cc:729] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.104980 15582 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:01.105083 15581 tablet_bootstrap.cc:654] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.106164 15582 tablet_bootstrap.cc:654] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.106437 15581 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:01.106537 15581 ts_tablet_manager.cc:1367] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:00:01.107170 15581 raft_consensus.cc:348] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.107280 15581 raft_consensus.cc:374] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.107312 15581 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.107435 15581 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.107718 15581 ts_tablet_manager.cc:1398] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.107882 15581 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:01.107928 15582 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:01.108000 15582 ts_tablet_manager.cc:1367] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:00:01.104681 15583 consensus_queue.cc:260] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.108649 15582 raft_consensus.cc:348] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.108765 15582 raft_consensus.cc:374] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.108786 15581 tablet_bootstrap.cc:654] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.108805 15582 raft_consensus.cc:729] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.108975 15582 consensus_queue.cc:260] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.109066 15583 ts_tablet_manager.cc:1398] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20240503 14:00:01.109413 15582 ts_tablet_manager.cc:1398] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:01.109464 15583 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:01.110320 15583 tablet_bootstrap.cc:654] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.111243 15582 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:01.111577 15583 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:01.111666 15583 ts_tablet_manager.cc:1367] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:00:01.112167 15583 raft_consensus.cc:348] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.112251 15583 raft_consensus.cc:374] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.112272 15583 raft_consensus.cc:729] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.112375 15583 consensus_queue.cc:260] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.112392 15582 tablet_bootstrap.cc:654] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.112668 15583 ts_tablet_manager.cc:1398] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:01.112807 15583 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:01.113672 15583 tablet_bootstrap.cc:654] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.113739 15582 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:01.113823 15582 ts_tablet_manager.cc:1367] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:01.114396 15582 raft_consensus.cc:348] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.114526 15582 raft_consensus.cc:374] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.114571 15582 raft_consensus.cc:729] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.114748 15582 consensus_queue.cc:260] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.115063 15583 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:01.115146 15583 ts_tablet_manager.cc:1367] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:01.115160 15582 ts_tablet_manager.cc:1398] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:01.115592 15583 raft_consensus.cc:348] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.115689 15583 raft_consensus.cc:374] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.115722 15583 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.115855 15583 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.116096 15583 ts_tablet_manager.cc:1398] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:01.116323 15581 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:01.116436 15581 ts_tablet_manager.cc:1367] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:00:01.116994 15581 raft_consensus.cc:348] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.117121 15581 raft_consensus.cc:374] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.117158 15581 raft_consensus.cc:729] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.117354 15581 consensus_queue.cc:260] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.117666 15581 ts_tablet_manager.cc:1398] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.128732 15588 raft_consensus.cc:482] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:01.128968 15588 raft_consensus.cc:504] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.130270 15588 leader_election.cc:288] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:01.135177 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a" is_pre_election: true
I20240503 14:00:01.135587 14988 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.135706 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:01.136015 14856 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.136103 15055 leader_election.cc:302] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:01.136415 15588 raft_consensus.cc:2781] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:01.136494 15588 raft_consensus.cc:482] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:01.136535 15588 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.137451 15588 raft_consensus.cc:504] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.137823 15588 leader_election.cc:288] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:01.138173 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:01.138214 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:01.138281 14856 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.138340 14988 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.139493 14856 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.139557 14988 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.139793 15053 leader_election.cc:302] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6; no voters:
I20240503 14:00:01.139972 15588 raft_consensus.cc:2781] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:01.140210 15588 raft_consensus.cc:686] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:01.140597 15588 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.143833 14714 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } }
W20240503 14:00:01.156563 15167 tablet.cc:2369] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:00:01.217031 15035 tablet.cc:2369] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:01.222399 15588 raft_consensus.cc:482] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:01.222573 15588 raft_consensus.cc:504] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.223073 15588 leader_election.cc:288] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:01.223532 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:01.223680 14856 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.223604 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a" is_pre_election: true
I20240503 14:00:01.223796 14988 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.223985 15053 leader_election.cc:302] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6; no voters:
I20240503 14:00:01.224166 15588 raft_consensus.cc:2781] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:01.224241 15588 raft_consensus.cc:482] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:01.224289 15588 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.225142 15588 raft_consensus.cc:504] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.225494 15588 leader_election.cc:288] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:01.225934 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:01.225934 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:01.226097 14988 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.226097 14856 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.227106 14856 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.227106 14988 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.227404 15055 leader_election.cc:302] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:01.227617 15588 raft_consensus.cc:2781] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:01.227710 15588 raft_consensus.cc:686] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:01.227854 15588 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.229717 14714 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } }
W20240503 14:00:01.270807 14903 tablet.cc:2369] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:01.327809 15588 raft_consensus.cc:482] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:01.327963 15588 raft_consensus.cc:504] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.328415 15588 leader_election.cc:288] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:01.328938 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a" is_pre_election: true
I20240503 14:00:01.328938 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:01.329105 14856 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.329105 14988 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.329435 15053 leader_election.cc:302] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6; no voters:
I20240503 14:00:01.329635 15588 raft_consensus.cc:2781] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:01.329700 15588 raft_consensus.cc:482] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:01.329726 15588 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.330664 15588 raft_consensus.cc:504] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.331027 15588 leader_election.cc:288] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:01.331378 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:01.331378 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:01.331498 14856 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.331539 14988 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.332283 14988 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.332357 14856 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.332636 15053 leader_election.cc:302] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6; no voters:
I20240503 14:00:01.332792 15588 raft_consensus.cc:2781] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:01.332882 15588 raft_consensus.cc:686] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:01.333014 15588 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.334930 14714 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } }
W20240503 14:00:01.482782 15605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:01.483060 15605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:01.486372 15605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:00:01.584985 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:01.584985 14987 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:01.584985 14988 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:01.586058 15588 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.586506 14856 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:01.586515 15588 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.586022 15601 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.587330 15601 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.595016 15631 mvcc.cc:205] Tried to move back new op lower bound from 7023594707286052864 to 7023594705830891520. Current Snapshot: MvccSnapshot[applied={T|T < 7023594707286052864}]
I20240503 14:00:01.595176 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:01.595698 15633 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.598709 14986 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:01.599372 15638 mvcc.cc:205] Tried to move back new op lower bound from 7023594707286052864 to 7023594705830891520. Current Snapshot: MvccSnapshot[applied={T|T < 7023594707286052864}]
I20240503 14:00:01.600968 15588 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.630430 15648 mvcc.cc:205] Tried to move back new op lower bound from 7023594707286052864 to 7023594705830891520. Current Snapshot: MvccSnapshot[applied={T|T < 7023594707286052864}]
INSERT report
rows total: 2000
time total: 119.351 ms
time per row: 0.0596755 ms
I20240503 14:00:01.659808 14710 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39186:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:00:01.660387 14710 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:01.666751 14835 tablet_service.cc:1467] Processing CreateTablet for tablet ce6f858488e54943b56317d074f939e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.667016 15100 tablet_service.cc:1467] Processing CreateTablet for tablet 80ab8aa2328344f58eedbe280175cd59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.667158 14835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f858488e54943b56317d074f939e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.667294 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ab8aa2328344f58eedbe280175cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.667419 15098 tablet_service.cc:1467] Processing CreateTablet for tablet ce6f858488e54943b56317d074f939e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.667651 15098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f858488e54943b56317d074f939e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.669766 14968 tablet_service.cc:1467] Processing CreateTablet for tablet ce6f858488e54943b56317d074f939e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.670100 14968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f858488e54943b56317d074f939e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.670444 14834 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee758d531074f1b98691319ecdcf21f (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.670454 14967 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee758d531074f1b98691319ecdcf21f (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.670668 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee758d531074f1b98691319ecdcf21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.670671 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee758d531074f1b98691319ecdcf21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.672461 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 80ab8aa2328344f58eedbe280175cd59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.672684 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ab8aa2328344f58eedbe280175cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.673295 14835 tablet_service.cc:1467] Processing CreateTablet for tablet 80ab8aa2328344f58eedbe280175cd59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.673427 15659 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:01.673444 15100 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee758d531074f1b98691319ecdcf21f (DEFAULT_TABLE table=rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:01.673619 14835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ab8aa2328344f58eedbe280175cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.673676 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee758d531074f1b98691319ecdcf21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:01.674345 15659 tablet_bootstrap.cc:654] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.674690 15658 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:01.675019 15660 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:01.675647 15658 tablet_bootstrap.cc:654] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.675808 15660 tablet_bootstrap.cc:654] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.676707 15658 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:01.676805 15658 ts_tablet_manager.cc:1367] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:01.676894 15660 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:01.676977 15660 ts_tablet_manager.cc:1367] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:01.677238 15658 raft_consensus.cc:348] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.677358 15658 raft_consensus.cc:374] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.677392 15658 raft_consensus.cc:729] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.677407 15660 raft_consensus.cc:348] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.677496 15660 raft_consensus.cc:374] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.677532 15660 raft_consensus.cc:729] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.677521 15658 consensus_queue.cc:260] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.677654 15660 consensus_queue.cc:260] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.677798 15658 ts_tablet_manager.cc:1398] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:01.677937 15660 ts_tablet_manager.cc:1398] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.678005 15658 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:01.678767 15660 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:01.679656 15660 tablet_bootstrap.cc:654] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.680413 15658 tablet_bootstrap.cc:654] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.681108 15660 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:01.681192 15660 ts_tablet_manager.cc:1367] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:01.681556 15659 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:01.681658 15659 ts_tablet_manager.cc:1367] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:00:01.681663 15660 raft_consensus.cc:348] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.681780 15660 raft_consensus.cc:374] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.681818 15660 raft_consensus.cc:729] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.681959 15660 consensus_queue.cc:260] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.682189 15659 raft_consensus.cc:348] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.682273 15660 ts_tablet_manager.cc:1398] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:01.682302 15659 raft_consensus.cc:374] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.682337 15659 raft_consensus.cc:729] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.682425 15660 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:01.682467 15659 consensus_queue.cc:260] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.683320 15660 tablet_bootstrap.cc:654] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.683326 15659 ts_tablet_manager.cc:1398] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:01.683481 15659 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:01.684482 15659 tablet_bootstrap.cc:654] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.685400 15660 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:01.685490 15660 ts_tablet_manager.cc:1367] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:00:01.685621 15658 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:01.685714 15658 ts_tablet_manager.cc:1367] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:00:01.685853 15659 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:01.685927 15660 raft_consensus.cc:348] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.686004 15659 ts_tablet_manager.cc:1367] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:00:01.686062 15660 raft_consensus.cc:374] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.686103 15660 raft_consensus.cc:729] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.686179 15658 raft_consensus.cc:348] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.686287 15658 raft_consensus.cc:374] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.686251 15660 consensus_queue.cc:260] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.686323 15658 raft_consensus.cc:729] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.686439 15658 consensus_queue.cc:260] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.686559 15660 ts_tablet_manager.cc:1398] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.686725 15658 ts_tablet_manager.cc:1398] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:01.686821 15659 raft_consensus.cc:348] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.686929 15658 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:01.686931 15659 raft_consensus.cc:374] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.686966 15659 raft_consensus.cc:729] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.687093 15659 consensus_queue.cc:260] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.687312 15659 ts_tablet_manager.cc:1398] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:01.687458 15659 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:01.687716 15658 tablet_bootstrap.cc:654] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.688302 15659 tablet_bootstrap.cc:654] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:01.689473 15658 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:01.689558 15658 ts_tablet_manager.cc:1367] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:00:01.689692 15659 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:01.689764 15659 ts_tablet_manager.cc:1367] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:01.690013 15658 raft_consensus.cc:348] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.690135 15658 raft_consensus.cc:374] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.690176 15658 raft_consensus.cc:729] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.690196 15659 raft_consensus.cc:348] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.690291 15659 raft_consensus.cc:374] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:01.690332 15659 raft_consensus.cc:729] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:01.690312 15658 consensus_queue.cc:260] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.690459 15659 consensus_queue.cc:260] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.690589 15658 ts_tablet_manager.cc:1398] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.690761 15659 ts_tablet_manager.cc:1398] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:01.774206 15634 raft_consensus.cc:482] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:01.774349 15634 raft_consensus.cc:504] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.774863 15634 leader_election.cc:288] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449)
I20240503 14:00:01.775315 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ab8aa2328344f58eedbe280175cd59" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:01.775319 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ab8aa2328344f58eedbe280175cd59" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a" is_pre_election: true
I20240503 14:00:01.775482 14988 raft_consensus.cc:2454] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.775501 14856 raft_consensus.cc:2454] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 0.
I20240503 14:00:01.775799 15053 leader_election.cc:302] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6; no voters:
I20240503 14:00:01.776000 15634 raft_consensus.cc:2781] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:01.776063 15634 raft_consensus.cc:482] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:01.776110 15634 raft_consensus.cc:3037] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.777007 15634 raft_consensus.cc:504] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.777371 15634 leader_election.cc:288] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Requested vote from peers 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449)
I20240503 14:00:01.777757 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ab8aa2328344f58eedbe280175cd59" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:01.777776 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ab8aa2328344f58eedbe280175cd59" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:01.777878 14988 raft_consensus.cc:3037] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.777884 14856 raft_consensus.cc:3037] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.778715 14856 raft_consensus.cc:2454] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.778719 14988 raft_consensus.cc:2454] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 1.
I20240503 14:00:01.779029 15053 leader_election.cc:302] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6; no voters:
I20240503 14:00:01.779237 15634 raft_consensus.cc:2781] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:01.779335 15634 raft_consensus.cc:686] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:01.779487 15634 consensus_queue.cc:237] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.781394 14713 catalog_manager.cc:5484] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:01.787767 15598 raft_consensus.cc:482] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:01.787914 15598 raft_consensus.cc:504] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.789026 15598 leader_election.cc:288] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:01.792838 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f858488e54943b56317d074f939e6" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:01.792842 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f858488e54943b56317d074f939e6" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6" is_pre_election: true
I20240503 14:00:01.793017 14856 raft_consensus.cc:2454] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:01.793078 15120 raft_consensus.cc:2454] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:01.793406 14921 leader_election.cc:302] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:01.793689 15598 raft_consensus.cc:2781] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:01.793749 15598 raft_consensus.cc:482] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:01.793776 15598 raft_consensus.cc:3037] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.794718 15598 raft_consensus.cc:504] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.795092 15598 leader_election.cc:288] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:01.795487 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f858488e54943b56317d074f939e6" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:01.795513 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f858488e54943b56317d074f939e6" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:01.795610 14856 raft_consensus.cc:3037] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.795639 15120 raft_consensus.cc:3037] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.796413 14856 raft_consensus.cc:2454] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:01.796464 15120 raft_consensus.cc:2454] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:01.796751 14923 leader_election.cc:302] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:01.796942 15598 raft_consensus.cc:2781] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:01.797168 15598 raft_consensus.cc:686] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Becoming Leader. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:01.797520 15598 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.800029 14713 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89314f1a84d8474dba0db8615c043d1a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.830614 15591 raft_consensus.cc:482] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:01.830803 15591 raft_consensus.cc:504] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.831955 15591 leader_election.cc:288] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:01.835829 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee758d531074f1b98691319ecdcf21f" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6" is_pre_election: true
I20240503 14:00:01.835862 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee758d531074f1b98691319ecdcf21f" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a" is_pre_election: true
I20240503 14:00:01.836009 14988 raft_consensus.cc:2454] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 0.
I20240503 14:00:01.836016 15120 raft_consensus.cc:2454] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 0.
I20240503 14:00:01.836380 14791 leader_election.cc:302] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:01.836650 15591 raft_consensus.cc:2781] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:01.836712 15591 raft_consensus.cc:482] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:01.836750 15591 raft_consensus.cc:3037] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.837620 15591 raft_consensus.cc:504] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.838011 15591 leader_election.cc:288] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 1 election: Requested vote from peers 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:01.838394 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee758d531074f1b98691319ecdcf21f" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:01.838461 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee758d531074f1b98691319ecdcf21f" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:01.838516 14988 raft_consensus.cc:3037] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.838595 15120 raft_consensus.cc:3037] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:01.839308 14988 raft_consensus.cc:2454] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 1.
I20240503 14:00:01.839380 15120 raft_consensus.cc:2454] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 1.
I20240503 14:00:01.839664 14791 leader_election.cc:302] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:01.839854 15591 raft_consensus.cc:2781] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:01.839948 15591 raft_consensus.cc:686] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEADER]: Becoming Leader. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Running, Role: LEADER
I20240503 14:00:01.840313 15591 consensus_queue.cc:237] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:01.842867 14713 catalog_manager.cc:5484] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129). New cstate: current_term: 1 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:01.980588 15680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:01.980921 15680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:01.984244 15680 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:00:02.078740 14856 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.079246 15633 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.080001 14855 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.080627 14988 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.080711 15598 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.081545 15633 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.082600 15120 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.085043 15674 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.090178 14987 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.091069 15119 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.092876 15591 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.093106 15595 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.874 ms
time per row: 0.0629372 ms
I20240503 14:00:02.155365 14712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39186:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:00:02.155947 14712 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:02.161589 14835 tablet_service.cc:1467] Processing CreateTablet for tablet 78ad1e99b2b94ec980a853010482066e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.161751 14967 tablet_service.cc:1467] Processing CreateTablet for tablet bc5e78a9d4b94f79be8c01753d403f58 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.161934 14835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ad1e99b2b94ec980a853010482066e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.161962 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 78ad1e99b2b94ec980a853010482066e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.162063 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5e78a9d4b94f79be8c01753d403f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.162207 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ad1e99b2b94ec980a853010482066e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.163544 14968 tablet_service.cc:1467] Processing CreateTablet for tablet df230c00c4b34649af138b689a18ac16 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.163558 15100 tablet_service.cc:1467] Processing CreateTablet for tablet 78ad1e99b2b94ec980a853010482066e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.163767 14968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df230c00c4b34649af138b689a18ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.163841 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ad1e99b2b94ec980a853010482066e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.164060 15099 tablet_service.cc:1467] Processing CreateTablet for tablet df230c00c4b34649af138b689a18ac16 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.164268 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df230c00c4b34649af138b689a18ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.165693 15098 tablet_service.cc:1467] Processing CreateTablet for tablet bc5e78a9d4b94f79be8c01753d403f58 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.165737 15658 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:02.165802 15660 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:02.165921 15098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5e78a9d4b94f79be8c01753d403f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.166663 15658 tablet_bootstrap.cc:654] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.166709 15660 tablet_bootstrap.cc:654] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.168370 15659 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:02.168569 14834 tablet_service.cc:1467] Processing CreateTablet for tablet bc5e78a9d4b94f79be8c01753d403f58 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.168762 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5e78a9d4b94f79be8c01753d403f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.169173 15659 tablet_bootstrap.cc:654] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.170184 15659 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:02.170269 15659 ts_tablet_manager.cc:1367] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:02.170531 14836 tablet_service.cc:1467] Processing CreateTablet for tablet df230c00c4b34649af138b689a18ac16 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:02.170738 14836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df230c00c4b34649af138b689a18ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.170938 15659 raft_consensus.cc:348] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.171033 15659 raft_consensus.cc:374] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.171061 15659 raft_consensus.cc:729] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.171181 15659 consensus_queue.cc:260] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.171429 15659 ts_tablet_manager.cc:1398] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:02.171600 15659 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:02.172329 15659 tablet_bootstrap.cc:654] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.173157 15658 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:02.173189 15660 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:02.173267 15658 ts_tablet_manager.cc:1367] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:00:02.173280 15660 ts_tablet_manager.cc:1367] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:00:02.173671 15658 raft_consensus.cc:348] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.173767 15658 raft_consensus.cc:374] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.173801 15658 raft_consensus.cc:729] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.173771 15660 raft_consensus.cc:348] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.173884 15660 raft_consensus.cc:374] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.173915 15660 raft_consensus.cc:729] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.173936 15658 consensus_queue.cc:260] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.174069 15660 consensus_queue.cc:260] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.174250 15658 ts_tablet_manager.cc:1398] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:02.174348 15660 ts_tablet_manager.cc:1398] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:02.174424 15658 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:02.174644 15660 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:02.175689 15660 tablet_bootstrap.cc:654] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.176359 15658 tablet_bootstrap.cc:654] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.177135 15659 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:02.177228 15659 ts_tablet_manager.cc:1367] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:00:02.177732 15659 raft_consensus.cc:348] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.177839 15659 raft_consensus.cc:374] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.177867 15659 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.178014 15659 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.178300 15659 ts_tablet_manager.cc:1398] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:02.178450 15659 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:02.179271 15658 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:02.179298 15659 tablet_bootstrap.cc:654] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.179363 15658 ts_tablet_manager.cc:1367] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:00:02.180037 15658 raft_consensus.cc:348] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.180157 15658 raft_consensus.cc:374] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.180197 15658 raft_consensus.cc:729] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.180348 15658 consensus_queue.cc:260] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.180444 15660 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:02.180552 15660 ts_tablet_manager.cc:1367] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:00:02.180672 15659 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea: No bootstrap required, opened a new log
I20240503 14:00:02.180790 15659 ts_tablet_manager.cc:1367] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:02.181020 15660 raft_consensus.cc:348] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.181133 15660 raft_consensus.cc:374] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.181169 15660 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.181288 15660 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.181525 15658 ts_tablet_manager.cc:1398] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:02.181562 15660 ts_tablet_manager.cc:1398] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 14:00:02.181458 15659 raft_consensus.cc:348] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.181689 15659 raft_consensus.cc:374] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.181730 15660 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:02.181762 15659 raft_consensus.cc:729] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.181841 15658 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:02.182018 15659 consensus_queue.cc:260] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.182346 15659 ts_tablet_manager.cc:1398] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:02.182801 15658 tablet_bootstrap.cc:654] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.182956 15660 tablet_bootstrap.cc:654] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.183913 15658 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: No bootstrap required, opened a new log
I20240503 14:00:02.183955 15660 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: No bootstrap required, opened a new log
I20240503 14:00:02.184022 15658 ts_tablet_manager.cc:1367] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:02.184168 15660 ts_tablet_manager.cc:1367] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:02.184478 15658 raft_consensus.cc:348] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.184594 15658 raft_consensus.cc:374] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.184626 15658 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.184705 15660 raft_consensus.cc:348] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.184748 15658 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.184806 15660 raft_consensus.cc:374] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.184834 15660 raft_consensus.cc:729] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.184927 15660 consensus_queue.cc:260] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.185029 15658 ts_tablet_manager.cc:1398] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:02.185168 15660 ts_tablet_manager.cc:1398] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:02.269313 15598 raft_consensus.cc:482] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:02.269490 15598 raft_consensus.cc:504] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.269938 15598 leader_election.cc:288] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:02.270398 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:02.270435 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6" is_pre_election: true
I20240503 14:00:02.270586 15120 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:02.270586 14856 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:02.270921 14923 leader_election.cc:302] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:02.271148 15598 raft_consensus.cc:2781] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:02.271227 15598 raft_consensus.cc:482] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:02.271263 15598 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.272349 15598 raft_consensus.cc:504] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.272888 15598 leader_election.cc:288] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:02.273320 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:02.273386 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:02.273458 14856 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.273526 15120 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.274497 14856 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:02.274499 15120 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:02.274850 14923 leader_election.cc:302] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:02.275081 15598 raft_consensus.cc:2781] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:02.275179 15598 raft_consensus.cc:686] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Becoming Leader. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:02.275321 15598 consensus_queue.cc:237] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.277271 14712 catalog_manager.cc:5484] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89314f1a84d8474dba0db8615c043d1a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.281011 15598 raft_consensus.cc:482] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:02.281145 15598 raft_consensus.cc:504] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.281569 15598 leader_election.cc:288] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:02.282016 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ad1e99b2b94ec980a853010482066e" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6" is_pre_election: true
I20240503 14:00:02.282019 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ad1e99b2b94ec980a853010482066e" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:02.282182 15120 raft_consensus.cc:2454] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:02.282182 14856 raft_consensus.cc:2454] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:02.282476 14923 leader_election.cc:302] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:02.282752 15598 raft_consensus.cc:2781] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:02.282829 15598 raft_consensus.cc:482] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:02.282874 15598 raft_consensus.cc:3037] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.283919 15598 raft_consensus.cc:504] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.284350 15598 leader_election.cc:288] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:02.284808 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ad1e99b2b94ec980a853010482066e" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:02.284898 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ad1e99b2b94ec980a853010482066e" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:02.284979 15120 raft_consensus.cc:3037] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.285012 14856 raft_consensus.cc:3037] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.286068 15120 raft_consensus.cc:2454] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:02.286068 14856 raft_consensus.cc:2454] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:02.286478 14923 leader_election.cc:302] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:02.286666 15598 raft_consensus.cc:2781] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:02.286772 15598 raft_consensus.cc:686] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Becoming Leader. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:02.286919 15598 consensus_queue.cc:237] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.288730 14712 catalog_manager.cc:5484] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89314f1a84d8474dba0db8615c043d1a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:02.449695 15598 raft_consensus.cc:482] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:02.449855 15598 raft_consensus.cc:504] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.450320 15598 leader_election.cc:288] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:02.450837 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6" is_pre_election: true
I20240503 14:00:02.450837 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea" is_pre_election: true
I20240503 14:00:02.450997 15120 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:02.450997 14856 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 0.
I20240503 14:00:02.451350 14923 leader_election.cc:302] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:02.451531 15598 raft_consensus.cc:2781] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:02.451603 15598 raft_consensus.cc:482] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:02.451629 15598 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.452461 15598 raft_consensus.cc:504] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.452832 15598 leader_election.cc:288] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:02.453258 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:02.453258 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:02.453402 15120 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.453410 14856 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.454473 14856 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:02.454473 15120 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 1.
I20240503 14:00:02.454849 14923 leader_election.cc:302] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:02.455030 15598 raft_consensus.cc:2781] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:02.455121 15598 raft_consensus.cc:686] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Becoming Leader. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:02.455276 15598 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } }
I20240503 14:00:02.457141 14712 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89314f1a84d8474dba0db8615c043d1a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:02.582787 15729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:02.583107 15729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:02.586454 15729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:00:02.665601 15120 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.666041 14856 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.665951 15119 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.666478 15598 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.666764 15598 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.666810 15674 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.669097 14855 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.671168 15598 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.672119 15118 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.672493 15674 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:02.674391 14854 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:02.674906 15674 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.928 ms
time per row: 0.058464 ms
I20240503 14:00:02.745476 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:32851
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:37509
--webserver_port=40767
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:00:02.857870 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:02.858193 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:02.858243 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:02.861441 15784 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:02.861526 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:02.861599 15784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 14:00:02.865522 15784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:32851
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=40767
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:02.866575 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:02.867761 15784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:02.877393 15789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.877799 15790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.878638 15792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:02.879397 15784 server_base.cc:1034] running on GCE node
I20240503 14:00:02.879921 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:02.880594 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:02.881798 15784 hybrid_clock.cc:648] HybridClock initialized: now 1714744802881767 us; error 45 us; skew 500 ppm
I20240503 14:00:02.884130 15784 webserver.cc:460] Webserver started at http://127.0.99.132:40767/ using document root <none> and password file <none>
I20240503 14:00:02.884757 15784 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:02.884831 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:02.888798 15784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:02.891155 15798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.892338 15784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:02.892460 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "35b5bdde0c49481e8d69060ad4d1d81d"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 14:00:02.892951 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:02.915562 15784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:02.916239 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:02.916419 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:02.917019 15784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:02.917968 15784 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:02.918056 15784 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.918107 15784 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:02.918133 15784 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.927134 15784 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:32851
I20240503 14:00:02.927167 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:32851 every 8 connection(s)
I20240503 14:00:02.928386 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:00:02.931654 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 15784
I20240503 14:00:02.932193 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:38635
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:37509
--webserver_port=38757
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:02.938967 15912 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 14:00:02.939236 15912 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:02.939852 15912 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 14:00:02.941313 14712 ts_manager.cc:194] Re-registered known tserver with Master: 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851)
I20240503 14:00:02.942112 14712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60639
W20240503 14:00:03.050504 15915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.050781 15915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.050827 15915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.054322 15915 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:03.054409 15915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.054483 15915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 14:00:03.058545 15915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:38635
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=38757
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.15915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.059677 15915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.060961 15915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.070672 15921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.071444 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.072149 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.073005 15915 server_base.cc:1034] running on GCE node
I20240503 14:00:03.073503 15915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.074254 15915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.075421 15915 hybrid_clock.cc:648] HybridClock initialized: now 1714744803075393 us; error 41 us; skew 500 ppm
I20240503 14:00:03.077564 15915 webserver.cc:460] Webserver started at http://127.0.99.134:38757/ using document root <none> and password file <none>
I20240503 14:00:03.078264 15915 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.078351 15915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.082273 15915 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:00:03.084621 15929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.085886 15915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:03.086050 15915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e90a2a2980f84d49a2532f143f71e753"
format_stamp: "Formatted at 2024-05-03 13:59:57 on dist-test-slave-6fwv"
I20240503 14:00:03.086577 15915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.103709 15915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.104396 15915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.104593 15915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.105170 15915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.106153 15915 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.106204 15915 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.106259 15915 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.106284 15915 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.115350 15915 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38635
I20240503 14:00:03.115378 16042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38635 every 8 connection(s)
I20240503 14:00:03.116482 15915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:00:03.118746 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 15915
I20240503 14:00:03.119133 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:46269
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:37509
--webserver_port=46035
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:03.126068 16043 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 14:00:03.126374 16043 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.127017 16043 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 14:00:03.128151 14712 ts_manager.cc:194] Re-registered known tserver with Master: e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635)
I20240503 14:00:03.128732 14712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:41367
W20240503 14:00:03.237666 16046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.237934 16046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.238029 16046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.241237 16046 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:03.241338 16046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.241428 16046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 14:00:03.245401 16046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:46269
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=46035
--tserver_master_addrs=127.0.99.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.16046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.246507 16046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.247745 16046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.257417 16051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.257920 16052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.258651 16054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.259481 16046 server_base.cc:1034] running on GCE node
I20240503 14:00:03.259984 16046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.260677 16046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.261842 16046 hybrid_clock.cc:648] HybridClock initialized: now 1714744803261809 us; error 55 us; skew 500 ppm
I20240503 14:00:03.263993 16046 webserver.cc:460] Webserver started at http://127.0.99.133:46035/ using document root <none> and password file <none>
I20240503 14:00:03.264638 16046 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.264751 16046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.269455 16046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:03.272284 16060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.273780 16046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:03.274014 16046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fb0b622a674141069043d816ad306f3e"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-6fwv"
I20240503 14:00:03.274747 16046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.302191 16046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.302901 16046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.303071 16046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.303664 16046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.304610 16046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.304659 16046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.304702 16046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.304716 16046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.313580 16046 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46269
I20240503 14:00:03.313598 16173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46269 every 8 connection(s)
I20240503 14:00:03.314852 16046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:00:03.315524 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 16046
I20240503 14:00:03.327659 16174 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37509
I20240503 14:00:03.327941 16174 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.328584 16174 heartbeater.cc:511] Master 127.0.99.190:37509 requested a full tablet report, sending...
I20240503 14:00:03.329711 14712 ts_manager.cc:194] Re-registered known tserver with Master: fb0b622a674141069043d816ad306f3e (127.0.99.133:46269)
I20240503 14:00:03.330353 14712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:38113
I20240503 14:00:03.528947 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.530063 14836 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:03.537431 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.537899 15970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.538760 15098 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.554258 14968 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.642822 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.646075 14968 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.647383 15970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.647646 15098 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.657997 14836 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:03.658201 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.780283 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.781769 15098 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.782618 14836 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:03.785264 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.786007 15970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.792856 14968 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.886288 15098 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.886438 14968 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:03.886948 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.887506 14836 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:03.888167 15970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.910041 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:03.942725 15120 consensus_queue.cc:237] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:03.944294 15912 heartbeater.cc:503] Master 127.0.99.190:37509 was elected leader, sending a full tablet report...
I20240503 14:00:03.944487 14987 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.944885 14856 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.945147 16329 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.945348 16328 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.947762 15588 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:03.948436 14987 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:03.948566 14856 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
W20240503 14:00:03.949813 15056 consensus_peers.cc:485] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 -> Peer 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851): Couldn't send request to peer 35b5bdde0c49481e8d69060ad4d1d81d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80ab8aa2328344f58eedbe280175cd59. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.951411 14713 catalog_manager.cc:5484] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:03.960196 15120 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:03.961808 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.962338 14987 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.962298 16328 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.962710 16328 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.964409 16329 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
W20240503 14:00:03.965957 15056 consensus_peers.cc:485] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d68b8332ce64eb2935deba765c3946a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.966759 14987 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:03.967056 14711 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:03.969959 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:03.975898 14987 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:03.977331 14856 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.977758 15120 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.978024 16351 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.978217 16216 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.980281 15674 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:03.980665 15120 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
W20240503 14:00:03.981877 14923 consensus_peers.cc:485] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a -> Peer fb0b622a674141069043d816ad306f3e (127.0.99.133:46269): Couldn't send request to peer fb0b622a674141069043d816ad306f3e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc5e78a9d4b94f79be8c01753d403f58. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.982359 14856 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:03.983577 14712 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:03.991385 14987 consensus_queue.cc:237] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:03.992878 15120 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.993149 14856 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:03.993440 16216 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:03.993804 16216 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:03.996845 14924 consensus_peers.cc:485] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a -> Peer 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851): Couldn't send request to peer 35b5bdde0c49481e8d69060ad4d1d81d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78ad1e99b2b94ec980a853010482066e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:03.996979 16216 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:03.997402 15120 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:03.997467 14855 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:03.999817 14711 catalog_manager.cc:5484] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:04.007359 15120 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.008942 14855 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:04.009042 15056 consensus_peers.cc:485] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb5245fe54de4741ada591ab9576e699. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:04.009370 15588 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.009464 14987 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:04.009840 15588 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.011310 16329 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.011801 14855 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.012219 14987 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.014708 14712 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:04.023798 14855 consensus_queue.cc:237] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.025478 14987 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:04.025874 15120 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:04.025954 15711 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.026291 16378 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.027931 15711 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.028787 15120 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.029067 14987 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
W20240503 14:00:04.031013 14791 consensus_peers.cc:485] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea -> Peer fb0b622a674141069043d816ad306f3e (127.0.99.133:46269): Couldn't send request to peer fb0b622a674141069043d816ad306f3e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ee758d531074f1b98691319ecdcf21f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:04.031744 14711 catalog_manager.cc:5484] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.039642 14987 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.041195 14855 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:04.041342 14924 consensus_peers.cc:485] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a -> Peer 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851): Couldn't send request to peer 35b5bdde0c49481e8d69060ad4d1d81d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce6f858488e54943b56317d074f939e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:04.041522 15120 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:04.041615 16351 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.041962 16216 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.043608 16351 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:04.044222 14855 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:04.045461 15120 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:04.047394 14712 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index -1 to 4, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:04.055963 15120 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.057471 14855 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:04.057533 14987 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:04.057770 15056 consensus_peers.cc:485] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 274cf23ceaaa4a049f8c35689918d7b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:04.057925 16328 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.058159 16329 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.059993 16329 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.060511 14855 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.063006 14711 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.064177 14987 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:04.070683 14987 consensus_queue.cc:237] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.072199 14855 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:04.072263 15120 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:04.072351 14923 consensus_peers.cc:485] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a -> Peer fb0b622a674141069043d816ad306f3e (127.0.99.133:46269): Couldn't send request to peer fb0b622a674141069043d816ad306f3e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df230c00c4b34649af138b689a18ac16. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:04.072620 16216 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.072824 16351 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:04.074863 15674 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.075294 14855 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.075649 15120 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.077880 14711 catalog_manager.cc:5484] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index -1 to 4, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:04.130928 16043 heartbeater.cc:503] Master 127.0.99.190:37509 was elected leader, sending a full tablet report...
I20240503 14:00:04.332089 16451 ts_tablet_manager.cc:902] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Initiating tablet copy from peer 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:04.332690 16174 heartbeater.cc:503] Master 127.0.99.190:37509 was elected leader, sending a full tablet report...
I20240503 14:00:04.333070 16451 tablet_copy_client.cc:308] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45503
I20240503 14:00:04.337373 15140 tablet_copy_service.cc:140] P 8614bbdff011473abb0f0cfce354d4a6: Received BeginTabletCopySession request for tablet 80ab8aa2328344f58eedbe280175cd59 from peer 35b5bdde0c49481e8d69060ad4d1d81d ({username='slave'} at 127.0.99.132:38985)
I20240503 14:00:04.337571 15140 tablet_copy_service.cc:161] P 8614bbdff011473abb0f0cfce354d4a6: Beginning new tablet copy session on tablet 80ab8aa2328344f58eedbe280175cd59 from peer 35b5bdde0c49481e8d69060ad4d1d81d at {username='slave'} at 127.0.99.132:38985: session id = 35b5bdde0c49481e8d69060ad4d1d81d-80ab8aa2328344f58eedbe280175cd59
I20240503 14:00:04.339686 15140 tablet_copy_source_session.cc:205] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.343744 16451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ab8aa2328344f58eedbe280175cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.350637 16451 tablet_copy_client.cc:783] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.350878 16451 tablet_copy_client.cc:647] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.353106 16451 tablet_copy_client.cc:523] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.355373 16451 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap starting.
I20240503 14:00:04.362825 16451 log.cc:824] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:04.374547 16493 ts_tablet_manager.cc:902] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:04.375358 16493 tablet_copy_client.cc:308] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:04.380290 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet bc5e78a9d4b94f79be8c01753d403f58 from peer fb0b622a674141069043d816ad306f3e ({username='slave'} at 127.0.99.133:39361)
I20240503 14:00:04.380455 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet bc5e78a9d4b94f79be8c01753d403f58 from peer fb0b622a674141069043d816ad306f3e at {username='slave'} at 127.0.99.133:39361: session id = fb0b622a674141069043d816ad306f3e-bc5e78a9d4b94f79be8c01753d403f58
I20240503 14:00:04.381799 15008 tablet_copy_source_session.cc:205] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.385805 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5e78a9d4b94f79be8c01753d403f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.388455 16451 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.388832 16451 tablet_bootstrap.cc:492] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap complete.
I20240503 14:00:04.389190 16451 ts_tablet_manager.cc:1367] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.008s
I20240503 14:00:04.392181 16451 raft_consensus.cc:348] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.392477 16451 raft_consensus.cc:729] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35b5bdde0c49481e8d69060ad4d1d81d, State: Initialized, Role: LEARNER
I20240503 14:00:04.392848 16451 consensus_queue.cc:260] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.393510 16493 tablet_copy_client.cc:783] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.393723 16493 tablet_copy_client.cc:647] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.393860 16451 ts_tablet_manager.cc:1398] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:00:04.394836 15140 tablet_copy_service.cc:342] P 8614bbdff011473abb0f0cfce354d4a6: Request end of tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-80ab8aa2328344f58eedbe280175cd59 received from {username='slave'} at 127.0.99.132:38985
I20240503 14:00:04.394933 15140 tablet_copy_service.cc:433] P 8614bbdff011473abb0f0cfce354d4a6: ending tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-80ab8aa2328344f58eedbe280175cd59 on tablet 80ab8aa2328344f58eedbe280175cd59 with peer 35b5bdde0c49481e8d69060ad4d1d81d
I20240503 14:00:04.396543 16493 tablet_copy_client.cc:523] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.398990 16493 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Bootstrap starting.
I20240503 14:00:04.404898 16493 log.cc:824] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:04.425493 16493 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.425894 16493 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Bootstrap complete.
I20240503 14:00:04.426276 16493 ts_tablet_manager.cc:1367] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Time spent bootstrapping tablet: real 0.027s user 0.027s sys 0.000s
I20240503 14:00:04.428578 16493 raft_consensus.cc:348] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.428838 16493 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb0b622a674141069043d816ad306f3e, State: Initialized, Role: LEARNER
I20240503 14:00:04.429183 16493 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.430083 16493 ts_tablet_manager.cc:1398] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:00:04.431135 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session fb0b622a674141069043d816ad306f3e-bc5e78a9d4b94f79be8c01753d403f58 received from {username='slave'} at 127.0.99.133:39361
I20240503 14:00:04.431244 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session fb0b622a674141069043d816ad306f3e-bc5e78a9d4b94f79be8c01753d403f58 on tablet bc5e78a9d4b94f79be8c01753d403f58 with peer fb0b622a674141069043d816ad306f3e
I20240503 14:00:04.479753 16512 ts_tablet_manager.cc:902] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Initiating tablet copy from peer 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:04.480664 16512 tablet_copy_client.cc:308] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45503
I20240503 14:00:04.484803 15140 tablet_copy_service.cc:140] P 8614bbdff011473abb0f0cfce354d4a6: Received BeginTabletCopySession request for tablet 6d68b8332ce64eb2935deba765c3946a from peer e90a2a2980f84d49a2532f143f71e753 ({username='slave'} at 127.0.99.134:59673)
I20240503 14:00:04.484926 15140 tablet_copy_service.cc:161] P 8614bbdff011473abb0f0cfce354d4a6: Beginning new tablet copy session on tablet 6d68b8332ce64eb2935deba765c3946a from peer e90a2a2980f84d49a2532f143f71e753 at {username='slave'} at 127.0.99.134:59673: session id = e90a2a2980f84d49a2532f143f71e753-6d68b8332ce64eb2935deba765c3946a
I20240503 14:00:04.485697 15140 tablet_copy_source_session.cc:205] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.489087 16512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d68b8332ce64eb2935deba765c3946a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.489362 16518 ts_tablet_manager.cc:902] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Initiating tablet copy from peer 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:04.489902 16518 tablet_copy_client.cc:308] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45503
I20240503 14:00:04.490469 15140 tablet_copy_service.cc:140] P 8614bbdff011473abb0f0cfce354d4a6: Received BeginTabletCopySession request for tablet eb5245fe54de4741ada591ab9576e699 from peer e90a2a2980f84d49a2532f143f71e753 ({username='slave'} at 127.0.99.134:59673)
I20240503 14:00:04.490586 15140 tablet_copy_service.cc:161] P 8614bbdff011473abb0f0cfce354d4a6: Beginning new tablet copy session on tablet eb5245fe54de4741ada591ab9576e699 from peer e90a2a2980f84d49a2532f143f71e753 at {username='slave'} at 127.0.99.134:59673: session id = e90a2a2980f84d49a2532f143f71e753-eb5245fe54de4741ada591ab9576e699
I20240503 14:00:04.490814 16493 ts_tablet_manager.cc:902] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: Initiating tablet copy from peer 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449)
I20240503 14:00:04.491274 16493 tablet_copy_client.cc:308] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43449
I20240503 14:00:04.491518 15140 tablet_copy_source_session.cc:205] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.492661 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5245fe54de4741ada591ab9576e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.495895 14876 tablet_copy_service.cc:140] P 6ecce61b6a854f02a65cfabb70a738ea: Received BeginTabletCopySession request for tablet 0ee758d531074f1b98691319ecdcf21f from peer fb0b622a674141069043d816ad306f3e ({username='slave'} at 127.0.99.133:34091)
I20240503 14:00:04.496042 16512 tablet_copy_client.cc:783] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.496059 14876 tablet_copy_service.cc:161] P 6ecce61b6a854f02a65cfabb70a738ea: Beginning new tablet copy session on tablet 0ee758d531074f1b98691319ecdcf21f from peer fb0b622a674141069043d816ad306f3e at {username='slave'} at 127.0.99.133:34091: session id = fb0b622a674141069043d816ad306f3e-0ee758d531074f1b98691319ecdcf21f
I20240503 14:00:04.496177 16518 tablet_copy_client.cc:783] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.496230 16512 tablet_copy_client.cc:647] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.496759 16518 tablet_copy_client.cc:647] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.497555 14876 tablet_copy_source_session.cc:205] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.498059 16512 tablet_copy_client.cc:523] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.498272 16518 tablet_copy_client.cc:523] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.499015 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee758d531074f1b98691319ecdcf21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.499946 16518 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Bootstrap starting.
I20240503 14:00:04.499971 16512 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Bootstrap starting.
I20240503 14:00:04.501386 16493 tablet_copy_client.cc:783] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.501551 16493 tablet_copy_client.cc:647] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.503244 16493 tablet_copy_client.cc:523] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.504647 16493 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: Bootstrap starting.
I20240503 14:00:04.505932 16518 log.cc:824] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:04.519107 16451 ts_tablet_manager.cc:902] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:04.519646 16451 tablet_copy_client.cc:308] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:04.524926 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet 78ad1e99b2b94ec980a853010482066e from peer 35b5bdde0c49481e8d69060ad4d1d81d ({username='slave'} at 127.0.99.132:54625)
I20240503 14:00:04.525065 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet 78ad1e99b2b94ec980a853010482066e from peer 35b5bdde0c49481e8d69060ad4d1d81d at {username='slave'} at 127.0.99.132:54625: session id = 35b5bdde0c49481e8d69060ad4d1d81d-78ad1e99b2b94ec980a853010482066e
I20240503 14:00:04.526394 15008 tablet_copy_source_session.cc:205] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.527536 16451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ad1e99b2b94ec980a853010482066e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.530272 16451 tablet_copy_client.cc:783] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.530786 16451 tablet_copy_client.cc:647] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.531819 16518 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.532311 16518 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Bootstrap complete.
I20240503 14:00:04.532691 16518 ts_tablet_manager.cc:1367] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.000s
I20240503 14:00:04.532701 16493 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.532956 16493 tablet_bootstrap.cc:492] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: Bootstrap complete.
I20240503 14:00:04.533108 16493 ts_tablet_manager.cc:1367] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240503 14:00:04.533404 16451 tablet_copy_client.cc:523] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.533452 16512 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.533699 16512 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Bootstrap complete.
I20240503 14:00:04.533742 16493 raft_consensus.cc:348] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.533843 16512 ts_tablet_manager.cc:1367] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Time spent bootstrapping tablet: real 0.034s user 0.032s sys 0.000s
I20240503 14:00:04.533860 16493 raft_consensus.cc:729] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb0b622a674141069043d816ad306f3e, State: Initialized, Role: LEARNER
I20240503 14:00:04.533965 16493 consensus_queue.cc:260] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.534736 16493 ts_tablet_manager.cc:1398] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:04.535238 16451 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap starting.
I20240503 14:00:04.535919 14876 tablet_copy_service.cc:342] P 6ecce61b6a854f02a65cfabb70a738ea: Request end of tablet copy session fb0b622a674141069043d816ad306f3e-0ee758d531074f1b98691319ecdcf21f received from {username='slave'} at 127.0.99.133:34091
I20240503 14:00:04.536018 14876 tablet_copy_service.cc:433] P 6ecce61b6a854f02a65cfabb70a738ea: ending tablet copy session fb0b622a674141069043d816ad306f3e-0ee758d531074f1b98691319ecdcf21f on tablet 0ee758d531074f1b98691319ecdcf21f with peer fb0b622a674141069043d816ad306f3e
I20240503 14:00:04.535951 16512 raft_consensus.cc:348] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.535950 16518 raft_consensus.cc:348] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.536288 16512 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Initialized, Role: LEARNER
I20240503 14:00:04.536303 16518 raft_consensus.cc:729] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Initialized, Role: LEARNER
I20240503 14:00:04.536662 16518 consensus_queue.cc:260] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.536662 16512 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.537715 16512 ts_tablet_manager.cc:1398] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:00:04.537811 16493 ts_tablet_manager.cc:902] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:04.538206 16493 tablet_copy_client.cc:308] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:04.538480 15140 tablet_copy_service.cc:342] P 8614bbdff011473abb0f0cfce354d4a6: Request end of tablet copy session e90a2a2980f84d49a2532f143f71e753-6d68b8332ce64eb2935deba765c3946a received from {username='slave'} at 127.0.99.134:59673
I20240503 14:00:04.538553 15140 tablet_copy_service.cc:433] P 8614bbdff011473abb0f0cfce354d4a6: ending tablet copy session e90a2a2980f84d49a2532f143f71e753-6d68b8332ce64eb2935deba765c3946a on tablet 6d68b8332ce64eb2935deba765c3946a with peer e90a2a2980f84d49a2532f143f71e753
I20240503 14:00:04.538836 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet df230c00c4b34649af138b689a18ac16 from peer fb0b622a674141069043d816ad306f3e ({username='slave'} at 127.0.99.133:39361)
I20240503 14:00:04.538955 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet df230c00c4b34649af138b689a18ac16 from peer fb0b622a674141069043d816ad306f3e at {username='slave'} at 127.0.99.133:39361: session id = fb0b622a674141069043d816ad306f3e-df230c00c4b34649af138b689a18ac16
I20240503 14:00:04.539920 15008 tablet_copy_source_session.cc:205] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.540861 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df230c00c4b34649af138b689a18ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.541074 16518 ts_tablet_manager.cc:1398] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:00:04.543107 16493 tablet_copy_client.cc:783] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.543272 16493 tablet_copy_client.cc:647] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.546226 15140 tablet_copy_service.cc:342] P 8614bbdff011473abb0f0cfce354d4a6: Request end of tablet copy session e90a2a2980f84d49a2532f143f71e753-eb5245fe54de4741ada591ab9576e699 received from {username='slave'} at 127.0.99.134:59673
I20240503 14:00:04.546327 15140 tablet_copy_service.cc:433] P 8614bbdff011473abb0f0cfce354d4a6: ending tablet copy session e90a2a2980f84d49a2532f143f71e753-eb5245fe54de4741ada591ab9576e699 on tablet eb5245fe54de4741ada591ab9576e699 with peer e90a2a2980f84d49a2532f143f71e753
I20240503 14:00:04.547621 16493 tablet_copy_client.cc:523] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.548887 16493 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: Bootstrap starting.
I20240503 14:00:04.557418 16451 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.557706 16451 tablet_bootstrap.cc:492] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap complete.
I20240503 14:00:04.557889 16451 ts_tablet_manager.cc:1367] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.000s
I20240503 14:00:04.558490 16451 raft_consensus.cc:348] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.558611 16451 raft_consensus.cc:729] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35b5bdde0c49481e8d69060ad4d1d81d, State: Initialized, Role: LEARNER
I20240503 14:00:04.558708 16451 consensus_queue.cc:260] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.558959 16451 ts_tablet_manager.cc:1398] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:00:04.559679 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-78ad1e99b2b94ec980a853010482066e received from {username='slave'} at 127.0.99.132:54625
I20240503 14:00:04.559787 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-78ad1e99b2b94ec980a853010482066e on tablet 78ad1e99b2b94ec980a853010482066e with peer 35b5bdde0c49481e8d69060ad4d1d81d
I20240503 14:00:04.570554 16493 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.570910 16493 tablet_bootstrap.cc:492] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: Bootstrap complete.
I20240503 14:00:04.571171 16493 ts_tablet_manager.cc:1367] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.003s
I20240503 14:00:04.571614 16518 ts_tablet_manager.cc:902] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Initiating tablet copy from peer 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:04.571774 16493 raft_consensus.cc:348] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.571926 16493 raft_consensus.cc:729] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb0b622a674141069043d816ad306f3e, State: Initialized, Role: LEARNER
I20240503 14:00:04.572064 16493 consensus_queue.cc:260] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:04.572160 16518 tablet_copy_client.cc:308] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45503
I20240503 14:00:04.572402 16493 ts_tablet_manager.cc:1398] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:04.572753 15140 tablet_copy_service.cc:140] P 8614bbdff011473abb0f0cfce354d4a6: Received BeginTabletCopySession request for tablet 274cf23ceaaa4a049f8c35689918d7b4 from peer e90a2a2980f84d49a2532f143f71e753 ({username='slave'} at 127.0.99.134:59673)
I20240503 14:00:04.572885 15140 tablet_copy_service.cc:161] P 8614bbdff011473abb0f0cfce354d4a6: Beginning new tablet copy session on tablet 274cf23ceaaa4a049f8c35689918d7b4 from peer e90a2a2980f84d49a2532f143f71e753 at {username='slave'} at 127.0.99.134:59673: session id = e90a2a2980f84d49a2532f143f71e753-274cf23ceaaa4a049f8c35689918d7b4
I20240503 14:00:04.573021 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session fb0b622a674141069043d816ad306f3e-df230c00c4b34649af138b689a18ac16 received from {username='slave'} at 127.0.99.133:39361
I20240503 14:00:04.573115 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session fb0b622a674141069043d816ad306f3e-df230c00c4b34649af138b689a18ac16 on tablet df230c00c4b34649af138b689a18ac16 with peer fb0b622a674141069043d816ad306f3e
I20240503 14:00:04.573930 15140 tablet_copy_source_session.cc:205] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.574939 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274cf23ceaaa4a049f8c35689918d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.577016 16518 tablet_copy_client.cc:783] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.577184 16518 tablet_copy_client.cc:647] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.578550 16518 tablet_copy_client.cc:523] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.579939 16518 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Bootstrap starting.
I20240503 14:00:04.604133 16518 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.604496 16518 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Bootstrap complete.
I20240503 14:00:04.604709 16518 ts_tablet_manager.cc:1367] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 14:00:04.605219 16518 raft_consensus.cc:348] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.605350 16518 raft_consensus.cc:729] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Initialized, Role: LEARNER
I20240503 14:00:04.605484 16518 consensus_queue.cc:260] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:04.605830 16518 ts_tablet_manager.cc:1398] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:04.606648 15140 tablet_copy_service.cc:342] P 8614bbdff011473abb0f0cfce354d4a6: Request end of tablet copy session e90a2a2980f84d49a2532f143f71e753-274cf23ceaaa4a049f8c35689918d7b4 received from {username='slave'} at 127.0.99.134:59673
I20240503 14:00:04.606734 15140 tablet_copy_service.cc:433] P 8614bbdff011473abb0f0cfce354d4a6: ending tablet copy session e90a2a2980f84d49a2532f143f71e753-274cf23ceaaa4a049f8c35689918d7b4 on tablet 274cf23ceaaa4a049f8c35689918d7b4 with peer e90a2a2980f84d49a2532f143f71e753
I20240503 14:00:04.616231 16451 ts_tablet_manager.cc:902] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:04.616662 16451 tablet_copy_client.cc:308] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:04.617201 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet ce6f858488e54943b56317d074f939e6 from peer 35b5bdde0c49481e8d69060ad4d1d81d ({username='slave'} at 127.0.99.132:54625)
I20240503 14:00:04.617313 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet ce6f858488e54943b56317d074f939e6 from peer 35b5bdde0c49481e8d69060ad4d1d81d at {username='slave'} at 127.0.99.132:54625: session id = 35b5bdde0c49481e8d69060ad4d1d81d-ce6f858488e54943b56317d074f939e6
I20240503 14:00:04.618317 15008 tablet_copy_source_session.cc:205] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:04.619920 16451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f858488e54943b56317d074f939e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:04.622249 16451 tablet_copy_client.cc:783] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:04.622416 16451 tablet_copy_client.cc:647] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:04.623919 16451 tablet_copy_client.cc:523] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:04.625370 16451 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap starting.
I20240503 14:00:04.648042 16451 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:04.648367 16451 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap complete.
I20240503 14:00:04.648568 16451 ts_tablet_manager.cc:1367] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240503 14:00:04.649098 16451 raft_consensus.cc:348] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.649245 16451 raft_consensus.cc:729] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35b5bdde0c49481e8d69060ad4d1d81d, State: Initialized, Role: LEARNER
I20240503 14:00:04.649361 16451 consensus_queue.cc:260] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:04.649657 16451 ts_tablet_manager.cc:1398] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:04.650195 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-ce6f858488e54943b56317d074f939e6 received from {username='slave'} at 127.0.99.132:54625
I20240503 14:00:04.650283 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-ce6f858488e54943b56317d074f939e6 on tablet ce6f858488e54943b56317d074f939e6 with peer 35b5bdde0c49481e8d69060ad4d1d81d
I20240503 14:00:04.818183 15866 raft_consensus.cc:1203] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:04.869798 15997 raft_consensus.cc:1203] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:04.915441 16128 raft_consensus.cc:1203] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:04.950325 15866 raft_consensus.cc:1203] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:05.043639 15997 raft_consensus.cc:1203] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:05.073836 16128 raft_consensus.cc:1203] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:05.110827 15997 raft_consensus.cc:1203] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:05.154987 16128 raft_consensus.cc:1203] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:05.215623 15866 raft_consensus.cc:1203] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:05.277770 16329 raft_consensus.cc:1050] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6: attempting to promote NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d to VOTER
I20240503 14:00:05.278337 16329 consensus_queue.cc:237] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:05.279980 14855 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.279992 14987 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.280119 15866 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.280659 16694 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.280930 16598 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.281107 16329 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.283787 16598 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.284703 14987 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.285789 14855 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.286567 15866 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.288283 14712 catalog_manager.cc:5484] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.292362 15120 consensus_queue.cc:237] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:05.293670 15866 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.293670 14987 raft_consensus.cc:1261] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.294207 16329 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.294431 16599 consensus_queue.cc:1035] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.296105 16694 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.296703 14987 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.296703 15866 raft_consensus.cc:2932] T 80ab8aa2328344f58eedbe280175cd59 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.298321 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80ab8aa2328344f58eedbe280175cd59 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:05.299216 14713 catalog_manager.cc:5484] T 80ab8aa2328344f58eedbe280175cd59 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.302028 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ab8aa2328344f58eedbe280175cd59 with delete_type TABLET_DATA_TOMBSTONED (TS 6ecce61b6a854f02a65cfabb70a738ea not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40884
I20240503 14:00:05.302711 16702 tablet_replica.cc:330] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: stopping tablet replica
I20240503 14:00:05.303117 16702 raft_consensus.cc:2229] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:05.303375 16702 raft_consensus.cc:2258] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:05.304540 16702 ts_tablet_manager.cc:1875] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:05.306712 16702 ts_tablet_manager.cc:1888] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:05.306820 16702 log.cc:1196] T 80ab8aa2328344f58eedbe280175cd59 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/80ab8aa2328344f58eedbe280175cd59
I20240503 14:00:05.307507 14698 catalog_manager.cc:4830] TS 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449): tablet 80ab8aa2328344f58eedbe280175cd59 (table rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]) successfully deleted
I20240503 14:00:05.353159 16351 raft_consensus.cc:1050] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER fb0b622a674141069043d816ad306f3e to VOTER
I20240503 14:00:05.353695 16351 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.355293 14855 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.355368 16128 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.355427 15120 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.355906 16704 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.356150 16351 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.356381 16645 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.359297 16645 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.360006 15120 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.360450 16128 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.362656 14855 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.363189 14708 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.386720 16599 raft_consensus.cc:1050] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: attempting to promote NON_VOTER e90a2a2980f84d49a2532f143f71e753 to VOTER
I20240503 14:00:05.387306 16599 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.389310 14855 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.389369 14987 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.389767 16694 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.390012 16329 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.390089 15997 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.390503 16694 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.393685 16694 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.394639 15997 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.395736 14987 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.397019 14713 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.397301 14855 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.404486 15120 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38008
I20240503 14:00:05.404685 15120 raft_consensus.cc:595] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.423496 16704 raft_consensus.cc:1050] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d to VOTER
I20240503 14:00:05.423995 16704 consensus_queue.cc:237] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:05.425457 15866 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.425488 15120 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.425593 14855 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.425911 16704 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.426133 16216 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.426430 16645 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.429352 16216 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.430039 15866 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.431528 14855 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.431746 15120 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.432130 14711 catalog_manager.cc:5484] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.437054 14987 consensus_queue.cc:237] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:05.439033 15120 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.439038 15866 raft_consensus.cc:1261] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.439460 16351 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.439652 16645 consensus_queue.cc:1035] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.441344 16704 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.441852 15120 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.443011 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78ad1e99b2b94ec980a853010482066e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:05.443065 15866 raft_consensus.cc:2932] T 78ad1e99b2b94ec980a853010482066e P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.443989 14708 catalog_manager.cc:5484] T 78ad1e99b2b94ec980a853010482066e P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.446772 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ad1e99b2b94ec980a853010482066e with delete_type TABLET_DATA_TOMBSTONED (TS 6ecce61b6a854f02a65cfabb70a738ea not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40884
I20240503 14:00:05.447024 16702 tablet_replica.cc:330] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: stopping tablet replica
I20240503 14:00:05.447178 16702 raft_consensus.cc:2229] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:05.447376 16702 raft_consensus.cc:2258] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:05.448232 16702 ts_tablet_manager.cc:1875] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:05.449254 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50834
I20240503 14:00:05.449425 14987 raft_consensus.cc:595] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.450495 16702 ts_tablet_manager.cc:1888] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:05.450573 16702 log.cc:1196] T 78ad1e99b2b94ec980a853010482066e P 6ecce61b6a854f02a65cfabb70a738ea: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/78ad1e99b2b94ec980a853010482066e
I20240503 14:00:05.451122 14698 catalog_manager.cc:4830] TS 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449): tablet 78ad1e99b2b94ec980a853010482066e (table rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]) successfully deleted
I20240503 14:00:05.486097 15120 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38070
I20240503 14:00:05.486284 15120 raft_consensus.cc:595] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.509285 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50878
I20240503 14:00:05.509452 14987 raft_consensus.cc:595] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.513420 16647 raft_consensus.cc:1050] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea: attempting to promote NON_VOTER fb0b622a674141069043d816ad306f3e to VOTER
I20240503 14:00:05.513919 16647 consensus_queue.cc:237] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.515507 15120 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.515497 16128 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.515679 14987 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.516085 16646 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.516309 16377 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.516626 16647 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.519529 16647 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.520120 15120 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.520488 16128 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.523255 14713 catalog_manager.cc:5484] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e reported cstate change: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.523336 14987 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.525094 16599 raft_consensus.cc:1050] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: attempting to promote NON_VOTER e90a2a2980f84d49a2532f143f71e753 to VOTER
I20240503 14:00:05.525576 16599 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.526901 14855 consensus_queue.cc:237] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.527698 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.528077 16329 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.528419 16128 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.528468 14987 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.528494 15997 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.528790 15120 raft_consensus.cc:1261] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.529174 16646 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.529371 16377 consensus_queue.cc:1035] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.529577 16329 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.530014 16329 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.531509 16377 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.532903 16598 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.533142 14698 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ee758d531074f1b98691319ecdcf21f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:05.534035 14987 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.534214 15997 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.534780 15119 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.535977 14713 catalog_manager.cc:5484] T 0ee758d531074f1b98691319ecdcf21f P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 5 to 6, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.536356 16128 raft_consensus.cc:2932] T 0ee758d531074f1b98691319ecdcf21f P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.538102 14708 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.539980 14856 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.542061 14968 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee758d531074f1b98691319ecdcf21f with delete_type TABLET_DATA_TOMBSTONED (TS 89314f1a84d8474dba0db8615c043d1a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50526
I20240503 14:00:05.542685 16835 tablet_replica.cc:330] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: stopping tablet replica
I20240503 14:00:05.543100 16835 raft_consensus.cc:2229] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:05.543359 16835 raft_consensus.cc:2258] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:05.544615 16835 ts_tablet_manager.cc:1875] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:05.547122 16835 ts_tablet_manager.cc:1888] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:05.547232 16835 log.cc:1196] T 0ee758d531074f1b98691319ecdcf21f P 89314f1a84d8474dba0db8615c043d1a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0ee758d531074f1b98691319ecdcf21f
I20240503 14:00:05.547788 14700 catalog_manager.cc:4830] TS 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623): tablet 0ee758d531074f1b98691319ecdcf21f (table rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]) successfully deleted
I20240503 14:00:05.555308 16329 raft_consensus.cc:1050] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: attempting to promote NON_VOTER e90a2a2980f84d49a2532f143f71e753 to VOTER
I20240503 14:00:05.555735 16329 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.557258 15997 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.557412 14856 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.557686 14987 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.557917 16598 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.558310 16598 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.558480 16329 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.561065 16599 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.561378 14987 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.562822 15997 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.563022 14856 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:05.564126 14712 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 4 to 5, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.571117 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38102
I20240503 14:00:05.571259 15119 raft_consensus.cc:595] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.580968 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38126
I20240503 14:00:05.581121 15119 raft_consensus.cc:595] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.591552 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50914
I20240503 14:00:05.591737 14987 raft_consensus.cc:595] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.598239 16351 raft_consensus.cc:1050] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER fb0b622a674141069043d816ad306f3e to VOTER
I20240503 14:00:05.598703 16351 consensus_queue.cc:237] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.600358 15119 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.600358 14856 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.600798 16704 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.600891 16128 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.601225 16704 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.601466 16216 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.603490 16216 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.603924 16128 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.604544 15119 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.606328 14708 catalog_manager.cc:5484] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.606804 14856 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:05.614210 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df230c00c4b34649af138b689a18ac16"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50940
I20240503 14:00:05.614378 14987 raft_consensus.cc:595] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.631964 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38138
I20240503 14:00:05.632131 15119 raft_consensus.cc:595] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.643172 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38152
I20240503 14:00:05.643347 15119 raft_consensus.cc:595] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.654336 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38170
I20240503 14:00:05.654515 15119 raft_consensus.cc:595] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.666329 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50968
I20240503 14:00:05.666491 14987 raft_consensus.cc:595] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.743813 16216 raft_consensus.cc:979] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: : Instructing follower 6ecce61b6a854f02a65cfabb70a738ea to start an election
I20240503 14:00:05.743948 16216 raft_consensus.cc:1067] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Signalling peer 6ecce61b6a854f02a65cfabb70a738ea to start an election
I20240503 14:00:05.744541 14856 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58"
dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
from {username='slave'} at 127.0.99.130:42397
I20240503 14:00:05.744717 14856 raft_consensus.cc:482] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:05.744773 14856 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.745604 14856 raft_consensus.cc:504] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.746064 14856 leader_election.cc:288] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 2 election: Requested vote from peers 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503), fb0b622a674141069043d816ad306f3e (127.0.99.133:46269)
I20240503 14:00:05.746466 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:05.746629 14987 raft_consensus.cc:3032] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:05.746582 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:05.746599 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5e78a9d4b94f79be8c01753d403f58" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0b622a674141069043d816ad306f3e"
I20240503 14:00:05.746677 14987 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:05.746728 15119 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.746793 16128 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.746850 14987 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.747061 14987 raft_consensus.cc:3037] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.747642 15119 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 2.
I20240503 14:00:05.747903 16128 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 2.
I20240503 14:00:05.747938 14987 raft_consensus.cc:2454] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 2.
I20240503 14:00:05.748266 14791 leader_election.cc:302] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:05.748450 16377 raft_consensus.cc:2781] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:05.748543 16377 raft_consensus.cc:686] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Becoming Leader. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Running, Role: LEADER
I20240503 14:00:05.748723 16377 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.750479 14711 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: term changed from 1 to 2, leader changed from 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) to 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129). New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:05.758241 16216 raft_consensus.cc:1050] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d to VOTER
I20240503 14:00:05.758720 16216 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:05.760238 15119 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.760246 15866 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.760291 14856 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:05.760680 16216 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.760896 16351 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.761097 16645 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:05.763375 16351 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.763971 15119 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.763937 14856 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.766660 14711 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.766770 15866 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.770466 14987 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:05.771679 15866 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.771804 15119 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:05.772049 16645 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.772276 16704 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:05.773931 16216 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.774441 15866 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.774552 15119 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.776512 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce6f858488e54943b56317d074f939e6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:05.776656 14708 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d reported cstate change: config changed from index 5 to 6, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:05.779232 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6f858488e54943b56317d074f939e6 with delete_type TABLET_DATA_TOMBSTONED (TS 6ecce61b6a854f02a65cfabb70a738ea not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40884
I20240503 14:00:05.779501 16702 tablet_replica.cc:330] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: stopping tablet replica
I20240503 14:00:05.779655 16702 raft_consensus.cc:2229] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:05.779842 16702 raft_consensus.cc:2258] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:05.780484 16702 ts_tablet_manager.cc:1875] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:05.782374 16702 ts_tablet_manager.cc:1888] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:05.782449 16702 log.cc:1196] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce6f858488e54943b56317d074f939e6
I20240503 14:00:05.782979 14698 catalog_manager.cc:4830] TS 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449): tablet ce6f858488e54943b56317d074f939e6 (table rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]) successfully deleted
I20240503 14:00:05.825996 16329 raft_consensus.cc:979] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: : Instructing follower e90a2a2980f84d49a2532f143f71e753 to start an election
I20240503 14:00:05.826136 16329 raft_consensus.cc:1067] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Signalling peer e90a2a2980f84d49a2532f143f71e753 to start an election
I20240503 14:00:05.826802 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "e90a2a2980f84d49a2532f143f71e753"
from {username='slave'} at 127.0.99.131:58951
I20240503 14:00:05.827015 15997 raft_consensus.cc:482] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:05.827126 15997 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.828074 15997 raft_consensus.cc:504] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.829236 15997 leader_election.cc:288] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [CANDIDATE]: Term 2 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:05.830176 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "e90a2a2980f84d49a2532f143f71e753" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:05.830333 15119 raft_consensus.cc:3032] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:05.830390 15119 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:05.830642 15119 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.830891 15119 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.832024 15119 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90a2a2980f84d49a2532f143f71e753 in term 2.
I20240503 14:00:05.833660 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "e90a2a2980f84d49a2532f143f71e753" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:05.833786 14856 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.833765 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "e90a2a2980f84d49a2532f143f71e753" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:05.833938 14987 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.834676 14856 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90a2a2980f84d49a2532f143f71e753 in term 2.
I20240503 14:00:05.834770 14987 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90a2a2980f84d49a2532f143f71e753 in term 2.
I20240503 14:00:05.835016 15930 leader_election.cc:302] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6, e90a2a2980f84d49a2532f143f71e753; no voters:
I20240503 14:00:05.835300 16535 raft_consensus.cc:2781] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:05.835579 16535 raft_consensus.cc:686] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Becoming Leader. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Running, Role: LEADER
I20240503 14:00:05.835916 16535 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.838433 14714 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 reported cstate change: term changed from 1 to 2, leader changed from 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) to e90a2a2980f84d49a2532f143f71e753 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:05.879057 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df230c00c4b34649af138b689a18ac16"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50994
I20240503 14:00:05.879225 14987 raft_consensus.cc:595] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.898193 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38190
I20240503 14:00:05.898358 15119 raft_consensus.cc:595] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.910317 16598 raft_consensus.cc:979] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: : Instructing follower 89314f1a84d8474dba0db8615c043d1a to start an election
I20240503 14:00:05.910444 16598 raft_consensus.cc:1067] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Signalling peer 89314f1a84d8474dba0db8615c043d1a to start an election
I20240503 14:00:05.910943 14987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
from {username='slave'} at 127.0.99.131:49621
I20240503 14:00:05.911123 14987 raft_consensus.cc:482] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:05.911185 14987 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.912268 14987 raft_consensus.cc:504] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.912894 14987 leader_election.cc:288] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 2 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503), e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635)
I20240503 14:00:05.913388 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:05.913529 15119 raft_consensus.cc:3032] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:05.913574 15119 raft_consensus.cc:729] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:05.913727 15119 consensus_queue.cc:260] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.913815 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:05.913936 15119 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.913950 14856 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.915019 14856 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 2.
I20240503 14:00:05.915035 15119 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 2.
I20240503 14:00:05.915729 14921 leader_election.cc:302] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:05.915949 16351 raft_consensus.cc:2781] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:05.916034 16351 raft_consensus.cc:686] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEADER]: Becoming Leader. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:05.916193 16351 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:05.918036 15997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699" candidate_uuid: "89314f1a84d8474dba0db8615c043d1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e90a2a2980f84d49a2532f143f71e753"
I20240503 14:00:05.918229 15997 raft_consensus.cc:3037] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.918398 14714 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: term changed from 1 to 2, leader changed from 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) to 89314f1a84d8474dba0db8615c043d1a (127.0.99.130). New cstate: current_term: 2 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:05.919126 15997 raft_consensus.cc:2454] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89314f1a84d8474dba0db8615c043d1a in term 2.
I20240503 14:00:05.919920 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb5245fe54de4741ada591ab9576e699"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38212
I20240503 14:00:05.920053 15119 raft_consensus.cc:595] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 14:00:05.920123 15119 raft_consensus.cc:603] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 14:00:05.938408 14987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df230c00c4b34649af138b689a18ac16"
dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51018
I20240503 14:00:05.938575 14987 raft_consensus.cc:595] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.949273 15119 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38234
I20240503 14:00:05.949440 15119 raft_consensus.cc:595] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:05.992486 16351 raft_consensus.cc:979] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: : Instructing follower 8614bbdff011473abb0f0cfce354d4a6 to start an election
I20240503 14:00:05.992605 16351 raft_consensus.cc:1067] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Signalling peer 8614bbdff011473abb0f0cfce354d4a6 to start an election
I20240503 14:00:05.993136 15119 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df230c00c4b34649af138b689a18ac16"
dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
from {username='slave'} at 127.0.99.130:44411
I20240503 14:00:05.993320 15119 raft_consensus.cc:482] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:05.993389 15119 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.994311 15119 raft_consensus.cc:504] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.994853 15119 leader_election.cc:288] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 2 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), fb0b622a674141069043d816ad306f3e (127.0.99.133:46269)
I20240503 14:00:05.995229 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:05.995280 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:05.995385 14856 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.995400 14987 raft_consensus.cc:3032] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:05.995442 14987 raft_consensus.cc:729] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Running, Role: LEADER
I20240503 14:00:05.995587 14987 consensus_queue.cc:260] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.995798 14987 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:05.996500 14856 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 2.
I20240503 14:00:05.996836 14987 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 2.
I20240503 14:00:05.997130 15055 leader_election.cc:302] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6, 89314f1a84d8474dba0db8615c043d1a; no voters:
I20240503 14:00:05.997383 16598 raft_consensus.cc:2781] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:05.997488 16598 raft_consensus.cc:686] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEADER]: Becoming Leader. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:05.997653 16598 consensus_queue.cc:237] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:05.999752 16128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df230c00c4b34649af138b689a18ac16" candidate_uuid: "8614bbdff011473abb0f0cfce354d4a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0b622a674141069043d816ad306f3e"
I20240503 14:00:05.999714 14714 catalog_manager.cc:5484] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: term changed from 1 to 2, leader changed from 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) to 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:05.999883 16128 raft_consensus.cc:3037] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:06.000680 16128 raft_consensus.cc:2454] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8614bbdff011473abb0f0cfce354d4a6 in term 2.
I20240503 14:00:06.110976 16329 raft_consensus.cc:979] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: : Instructing follower e90a2a2980f84d49a2532f143f71e753 to start an election
I20240503 14:00:06.111097 16329 raft_consensus.cc:1067] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Signalling peer e90a2a2980f84d49a2532f143f71e753 to start an election
I20240503 14:00:06.111567 15997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4"
dest_uuid: "e90a2a2980f84d49a2532f143f71e753"
from {username='slave'} at 127.0.99.131:58951
I20240503 14:00:06.111726 15997 raft_consensus.cc:482] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:06.111789 15997 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:06.112632 15997 raft_consensus.cc:504] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:06.113090 15997 leader_election.cc:288] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [CANDIDATE]: Term 2 election: Requested vote from peers 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449), 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503)
I20240503 14:00:06.113504 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "e90a2a2980f84d49a2532f143f71e753" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
I20240503 14:00:06.113535 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "e90a2a2980f84d49a2532f143f71e753" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:06.113519 14987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274cf23ceaaa4a049f8c35689918d7b4" candidate_uuid: "e90a2a2980f84d49a2532f143f71e753" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "89314f1a84d8474dba0db8615c043d1a"
I20240503 14:00:06.113656 15119 raft_consensus.cc:3032] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:06.113663 14856 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:06.113673 14987 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:06.113703 15119 raft_consensus.cc:729] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Running, Role: LEADER
I20240503 14:00:06.113854 15119 consensus_queue.cc:260] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:06.114086 15119 raft_consensus.cc:3037] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:06.114634 14856 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90a2a2980f84d49a2532f143f71e753 in term 2.
I20240503 14:00:06.114712 14987 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90a2a2980f84d49a2532f143f71e753 in term 2.
I20240503 14:00:06.115031 15119 raft_consensus.cc:2454] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90a2a2980f84d49a2532f143f71e753 in term 2.
I20240503 14:00:06.115072 15932 leader_election.cc:302] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 89314f1a84d8474dba0db8615c043d1a, e90a2a2980f84d49a2532f143f71e753; no voters:
I20240503 14:00:06.115401 16535 raft_consensus.cc:2781] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:06.115501 16535 raft_consensus.cc:686] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Becoming Leader. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Running, Role: LEADER
I20240503 14:00:06.115653 16535 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:06.117311 14714 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 reported cstate change: term changed from 1 to 2, leader changed from 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) to e90a2a2980f84d49a2532f143f71e753 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:06.143649 16128 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.144260 16377 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.219950 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.220531 16535 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.260213 15119 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.260740 16535 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.263651 14987 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.264407 16950 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.339099 15119 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.339592 16377 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.342703 14987 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.344069 16377 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.345026 14856 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:06.346458 15119 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.346454 16128 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.346839 16377 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.347023 17067 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.348729 17068 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.349205 15119 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.349205 16128 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.350739 14698 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc5e78a9d4b94f79be8c01753d403f58 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:06.351742 14708 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e reported cstate change: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.352843 15997 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.353287 16351 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.354187 14968 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5e78a9d4b94f79be8c01753d403f58 with delete_type TABLET_DATA_TOMBSTONED (TS 89314f1a84d8474dba0db8615c043d1a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50526
I20240503 14:00:06.354796 17070 tablet_replica.cc:330] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: stopping tablet replica
I20240503 14:00:06.355073 17070 raft_consensus.cc:2229] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:06.355195 17070 raft_consensus.cc:2258] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:06.355839 17070 ts_tablet_manager.cc:1875] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:06.357671 17070 ts_tablet_manager.cc:1888] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:06.357740 17070 log.cc:1196] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc5e78a9d4b94f79be8c01753d403f58
I20240503 14:00:06.358243 14700 catalog_manager.cc:4830] TS 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623): tablet bc5e78a9d4b94f79be8c01753d403f58 (table rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]) successfully deleted
I20240503 14:00:06.397745 16128 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.398428 16329 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.415987 15119 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.416486 16351 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.419562 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.420500 17086 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.492358 14856 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.492851 17063 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.507640 14856 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.508141 16329 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.511198 14987 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.511874 16598 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.512851 15119 consensus_queue.cc:237] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:06.514127 16128 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.514187 14856 raft_consensus.cc:1261] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 8614bbdff011473abb0f0cfce354d4a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.514602 17117 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.514747 16329 consensus_queue.cc:1035] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.516222 16329 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.517448 16128 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.518196 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df230c00c4b34649af138b689a18ac16 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:06.518144 14856 raft_consensus.cc:2932] T df230c00c4b34649af138b689a18ac16 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:06.519929 14708 catalog_manager.cc:5484] T df230c00c4b34649af138b689a18ac16 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 5 to 7, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "8614bbdff011473abb0f0cfce354d4a6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:06.522821 14968 tablet_service.cc:1514] Processing DeleteTablet for tablet df230c00c4b34649af138b689a18ac16 with delete_type TABLET_DATA_TOMBSTONED (TS 89314f1a84d8474dba0db8615c043d1a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50526
I20240503 14:00:06.523120 17070 tablet_replica.cc:330] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: stopping tablet replica
I20240503 14:00:06.523268 17070 raft_consensus.cc:2229] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:06.523397 17070 raft_consensus.cc:2258] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:06.524109 17070 ts_tablet_manager.cc:1875] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:06.526384 17070 ts_tablet_manager.cc:1888] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:06.526477 17070 log.cc:1196] T df230c00c4b34649af138b689a18ac16 P 89314f1a84d8474dba0db8615c043d1a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/df230c00c4b34649af138b689a18ac16
I20240503 14:00:06.527120 14700 catalog_manager.cc:4830] TS 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623): tablet df230c00c4b34649af138b689a18ac16 (table rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]) successfully deleted
I20240503 14:00:06.579635 14987 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.580184 17063 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.583173 15119 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:06.583894 16950 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:06.589751 15997 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:06.591104 14856 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.591269 14987 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.591643 16950 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.591827 17062 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.593506 17063 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.594044 14987 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.594022 14856 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.596168 14701 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 274cf23ceaaa4a049f8c35689918d7b4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:06.597366 14714 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.600492 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet 274cf23ceaaa4a049f8c35689918d7b4 with delete_type TABLET_DATA_TOMBSTONED (TS 8614bbdff011473abb0f0cfce354d4a6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37684
I20240503 14:00:06.601135 17121 tablet_replica.cc:330] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: stopping tablet replica
I20240503 14:00:06.601518 17121 raft_consensus.cc:2229] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:06.601693 17121 raft_consensus.cc:2258] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:06.602842 17121 ts_tablet_manager.cc:1875] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:06.605110 17121 ts_tablet_manager.cc:1888] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:06.605201 17121 log.cc:1196] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/274cf23ceaaa4a049f8c35689918d7b4
I20240503 14:00:06.605839 14700 catalog_manager.cc:4830] TS 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): tablet 274cf23ceaaa4a049f8c35689918d7b4 (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:06.816125 15997 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:06.817519 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.817515 14987 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.817937 17063 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.818199 17062 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.819869 17062 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.820341 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.820529 14987 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.821637 14701 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d68b8332ce64eb2935deba765c3946a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:06.822916 14708 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 reported cstate change: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:06.825816 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d68b8332ce64eb2935deba765c3946a with delete_type TABLET_DATA_TOMBSTONED (TS 8614bbdff011473abb0f0cfce354d4a6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37684
I20240503 14:00:06.826090 17121 tablet_replica.cc:330] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: stopping tablet replica
I20240503 14:00:06.826220 17121 raft_consensus.cc:2229] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:06.826330 17121 raft_consensus.cc:2258] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:06.826980 17121 ts_tablet_manager.cc:1875] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:06.829080 17121 ts_tablet_manager.cc:1888] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:06.829152 17121 log.cc:1196] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6d68b8332ce64eb2935deba765c3946a
I20240503 14:00:06.829592 14700 catalog_manager.cc:4830] TS 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): tablet 6d68b8332ce64eb2935deba765c3946a (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:06.910979 14987 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } }
I20240503 14:00:06.912297 15997 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.912276 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:06.912685 17086 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.912910 16216 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:06.915019 16351 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.915799 15997 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.915814 14856 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.918812 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb5245fe54de4741ada591ab9576e699 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:06.919202 14714 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 reported cstate change: config changed from index 5 to 7, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } }
I20240503 14:00:06.922005 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5245fe54de4741ada591ab9576e699 with delete_type TABLET_DATA_TOMBSTONED (TS 8614bbdff011473abb0f0cfce354d4a6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37684
I20240503 14:00:06.922250 17121 tablet_replica.cc:330] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: stopping tablet replica
I20240503 14:00:06.922384 17121 raft_consensus.cc:2229] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:06.922502 17121 raft_consensus.cc:2258] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:06.923368 17121 ts_tablet_manager.cc:1875] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:06.926482 17121 ts_tablet_manager.cc:1888] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:06.926566 17121 log.cc:1196] T eb5245fe54de4741ada591ab9576e699 P 8614bbdff011473abb0f0cfce354d4a6: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb5245fe54de4741ada591ab9576e699
I20240503 14:00:06.927197 14700 catalog_manager.cc:4830] TS 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): tablet eb5245fe54de4741ada591ab9576e699 (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:06.981012 14836 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:06.983573 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:06.983860 15098 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:06.985332 14968 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:06.995077 15970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:07.004148 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:07.097333 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:07.098286 14968 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:07.101962 15970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:07.104123 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:07.110472 15098 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:07.115351 14836 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:07.159116 15997 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } }
I20240503 14:00:07.159358 14987 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:07.160739 15997 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 14:00:07.161221 14924 consensus_peers.cc:485] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a -> Peer 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851): Couldn't send request to peer 35b5bdde0c49481e8d69060ad4d1d81d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb5245fe54de4741ada591ab9576e699. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.161218 14855 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:07.161219 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 14:00:07.161501 15932 consensus_peers.cc:485] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 -> Peer 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): Couldn't send request to peer 8614bbdff011473abb0f0cfce354d4a6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.161639 17086 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.162236 17063 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.162282 16216 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.162474 14987 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:07.162981 17063 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.163909 16351 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:07.164044 16950 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } } }
I20240503 14:00:07.164783 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } } }
I20240503 14:00:07.166183 14855 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:07.167156 14708 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 reported cstate change: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:07.167670 14712 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 7 to 8, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:07.172752 14987 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } } }
I20240503 14:00:07.176203 15996 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } } }
I20240503 14:00:07.179471 15996 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } }
I20240503 14:00:07.181002 14987 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:07.181223 14855 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 14:00:07.181622 15932 consensus_peers.cc:485] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 -> Peer 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): Couldn't send request to peer 8614bbdff011473abb0f0cfce354d4a6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.181856 16535 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.182267 17063 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.183813 16535 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } } }
I20240503 14:00:07.184007 14856 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:07.184389 14987 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } } }
I20240503 14:00:07.184412 14855 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } } }
I20240503 14:00:07.187433 15119 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:07.187686 16128 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:07.187927 17067 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:07.187960 14714 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 reported cstate change: config changed from index 7 to 8, NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) added. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:07.188908 17067 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:00:07.191897 14792 consensus_peers.cc:485] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc5e78a9d4b94f79be8c01753d403f58. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.191888 17067 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.195340 15119 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.195721 14712 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 7 to 8, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:07.197077 16128 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.203480 14855 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } }
I20240503 14:00:07.204183 14987 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } }
I20240503 14:00:07.205698 16128 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:07.205732 15120 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:00:07.205837 14792 consensus_peers.cc:485] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc5e78a9d4b94f79be8c01753d403f58. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.205732 15119 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:07.206164 16216 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:00:07.206260 14791 consensus_peers.cc:485] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea -> Peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623): Couldn't send request to peer 89314f1a84d8474dba0db8615c043d1a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.206324 15866 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:07.206558 17262 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:07.206777 17275 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:07.206876 16216 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:00:07.207917 14924 consensus_peers.cc:485] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce6f858488e54943b56317d074f939e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.208319 16645 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.208352 17262 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:07.208767 15120 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.211575 14708 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 6 to 7, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.211524 16128 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:07.211668 15866 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } }
I20240503 14:00:07.213529 15120 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:07.214552 14714 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 8 to 9, NON_VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:07.223855 14987 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } }
I20240503 14:00:07.225426 15120 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:07.225574 15866 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:07.225889 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:00:07.226235 14921 consensus_peers.cc:485] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a -> Peer 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449): Couldn't send request to peer 6ecce61b6a854f02a65cfabb70a738ea. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.226261 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:00:07.226644 14924 consensus_peers.cc:485] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a -> Peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): Couldn't send request to peer e90a2a2980f84d49a2532f143f71e753. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce6f858488e54943b56317d074f939e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:07.228282 16351 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } } }
I20240503 14:00:07.228713 15120 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } } }
I20240503 14:00:07.229265 15866 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } } }
I20240503 14:00:07.231874 14713 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d reported cstate change: config changed from index 7 to 8, NON_VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } } }
I20240503 14:00:07.629323 17358 ts_tablet_manager.cc:902] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:07.629880 17358 tablet_copy_client.cc:308] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:07.630501 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet eb5245fe54de4741ada591ab9576e699 from peer 35b5bdde0c49481e8d69060ad4d1d81d ({username='slave'} at 127.0.99.132:54625)
I20240503 14:00:07.630659 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet eb5245fe54de4741ada591ab9576e699 from peer 35b5bdde0c49481e8d69060ad4d1d81d at {username='slave'} at 127.0.99.132:54625: session id = 35b5bdde0c49481e8d69060ad4d1d81d-eb5245fe54de4741ada591ab9576e699
I20240503 14:00:07.631446 15008 tablet_copy_source_session.cc:205] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.632429 17358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5245fe54de4741ada591ab9576e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.634600 17358 tablet_copy_client.cc:783] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.634786 17358 tablet_copy_client.cc:647] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.636222 17358 tablet_copy_client.cc:523] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.637478 17358 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap starting.
I20240503 14:00:07.650789 17364 ts_tablet_manager.cc:902] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Initiating tablet copy from peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635)
I20240503 14:00:07.651613 17364 tablet_copy_client.cc:308] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:38635
I20240503 14:00:07.652484 16017 tablet_copy_service.cc:140] P e90a2a2980f84d49a2532f143f71e753: Received BeginTabletCopySession request for tablet 6d68b8332ce64eb2935deba765c3946a from peer 8614bbdff011473abb0f0cfce354d4a6 ({username='slave'} at 127.0.99.131:58951)
I20240503 14:00:07.652650 16017 tablet_copy_service.cc:161] P e90a2a2980f84d49a2532f143f71e753: Beginning new tablet copy session on tablet 6d68b8332ce64eb2935deba765c3946a from peer 8614bbdff011473abb0f0cfce354d4a6 at {username='slave'} at 127.0.99.131:58951: session id = 8614bbdff011473abb0f0cfce354d4a6-6d68b8332ce64eb2935deba765c3946a
I20240503 14:00:07.653833 16017 tablet_copy_source_session.cc:205] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.655189 17364 ts_tablet_manager.cc:1875] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:07.657570 17364 ts_tablet_manager.cc:1888] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:00:07.657802 17364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d68b8332ce64eb2935deba765c3946a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.658965 17358 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.659272 17358 tablet_bootstrap.cc:492] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: Bootstrap complete.
I20240503 14:00:07.659390 17364 tablet_copy_client.cc:783] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.659478 17358 ts_tablet_manager.cc:1367] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20240503 14:00:07.659605 17364 tablet_copy_client.cc:647] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.659955 17358 raft_consensus.cc:348] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:07.660082 17358 raft_consensus.cc:729] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 35b5bdde0c49481e8d69060ad4d1d81d, State: Initialized, Role: LEARNER
I20240503 14:00:07.660176 17358 consensus_queue.cc:260] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: true } }
I20240503 14:00:07.660424 17358 ts_tablet_manager.cc:1398] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:07.661332 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-eb5245fe54de4741ada591ab9576e699 received from {username='slave'} at 127.0.99.132:54625
I20240503 14:00:07.661437 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session 35b5bdde0c49481e8d69060ad4d1d81d-eb5245fe54de4741ada591ab9576e699 on tablet eb5245fe54de4741ada591ab9576e699 with peer 35b5bdde0c49481e8d69060ad4d1d81d
I20240503 14:00:07.662283 17364 tablet_copy_client.cc:523] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.664443 17364 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:07.691103 17364 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.691460 17364 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap complete.
I20240503 14:00:07.691676 17364 ts_tablet_manager.cc:1367] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.002s
I20240503 14:00:07.692101 17364 raft_consensus.cc:348] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } }
I20240503 14:00:07.692214 17364 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: LEARNER
I20240503 14:00:07.692314 17364 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } }
I20240503 14:00:07.692638 17364 ts_tablet_manager.cc:1398] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:07.693656 16017 tablet_copy_service.cc:342] P e90a2a2980f84d49a2532f143f71e753: Request end of tablet copy session 8614bbdff011473abb0f0cfce354d4a6-6d68b8332ce64eb2935deba765c3946a received from {username='slave'} at 127.0.99.131:58951
I20240503 14:00:07.693768 16017 tablet_copy_service.cc:433] P e90a2a2980f84d49a2532f143f71e753: ending tablet copy session 8614bbdff011473abb0f0cfce354d4a6-6d68b8332ce64eb2935deba765c3946a on tablet 6d68b8332ce64eb2935deba765c3946a with peer 8614bbdff011473abb0f0cfce354d4a6
I20240503 14:00:07.709570 17400 ts_tablet_manager.cc:902] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:07.710433 17400 tablet_copy_client.cc:308] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:07.711156 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet ce6f858488e54943b56317d074f939e6 from peer 6ecce61b6a854f02a65cfabb70a738ea ({username='slave'} at 127.0.99.129:39707)
I20240503 14:00:07.711270 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet ce6f858488e54943b56317d074f939e6 from peer 6ecce61b6a854f02a65cfabb70a738ea at {username='slave'} at 127.0.99.129:39707: session id = 6ecce61b6a854f02a65cfabb70a738ea-ce6f858488e54943b56317d074f939e6
I20240503 14:00:07.712040 15008 tablet_copy_source_session.cc:205] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.713152 17400 ts_tablet_manager.cc:1875] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:07.715682 17400 ts_tablet_manager.cc:1888] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:00:07.715885 17400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f858488e54943b56317d074f939e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.717072 17400 tablet_copy_client.cc:783] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.717245 17400 tablet_copy_client.cc:647] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.718813 17400 tablet_copy_client.cc:523] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.720404 17400 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap starting.
I20240503 14:00:07.735338 17364 ts_tablet_manager.cc:902] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Initiating tablet copy from peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635)
I20240503 14:00:07.735914 17364 tablet_copy_client.cc:308] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:38635
I20240503 14:00:07.736526 16017 tablet_copy_service.cc:140] P e90a2a2980f84d49a2532f143f71e753: Received BeginTabletCopySession request for tablet 274cf23ceaaa4a049f8c35689918d7b4 from peer 8614bbdff011473abb0f0cfce354d4a6 ({username='slave'} at 127.0.99.131:58951)
I20240503 14:00:07.736655 16017 tablet_copy_service.cc:161] P e90a2a2980f84d49a2532f143f71e753: Beginning new tablet copy session on tablet 274cf23ceaaa4a049f8c35689918d7b4 from peer 8614bbdff011473abb0f0cfce354d4a6 at {username='slave'} at 127.0.99.131:58951: session id = 8614bbdff011473abb0f0cfce354d4a6-274cf23ceaaa4a049f8c35689918d7b4
I20240503 14:00:07.737416 16017 tablet_copy_source_session.cc:205] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.738189 17364 ts_tablet_manager.cc:1875] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:07.740602 17364 ts_tablet_manager.cc:1888] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:00:07.740805 17364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274cf23ceaaa4a049f8c35689918d7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.742053 17364 tablet_copy_client.cc:783] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.742204 17364 tablet_copy_client.cc:647] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.743691 17364 tablet_copy_client.cc:523] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.744913 17364 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap starting.
I20240503 14:00:07.747390 17400 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.747702 17400 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Bootstrap complete.
I20240503 14:00:07.747900 17400 ts_tablet_manager.cc:1367] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20240503 14:00:07.748421 17400 raft_consensus.cc:348] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } }
I20240503 14:00:07.748594 17400 raft_consensus.cc:729] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Initialized, Role: LEARNER
I20240503 14:00:07.748678 17400 consensus_queue.cc:260] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } }
I20240503 14:00:07.749011 17400 ts_tablet_manager.cc:1398] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:07.749725 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session 6ecce61b6a854f02a65cfabb70a738ea-ce6f858488e54943b56317d074f939e6 received from {username='slave'} at 127.0.99.129:39707
I20240503 14:00:07.749822 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session 6ecce61b6a854f02a65cfabb70a738ea-ce6f858488e54943b56317d074f939e6 on tablet ce6f858488e54943b56317d074f939e6 with peer 6ecce61b6a854f02a65cfabb70a738ea
I20240503 14:00:07.765229 17364 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.765543 17364 tablet_bootstrap.cc:492] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Bootstrap complete.
I20240503 14:00:07.765754 17364 ts_tablet_manager.cc:1367] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240503 14:00:07.766269 17364 raft_consensus.cc:348] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } }
I20240503 14:00:07.766391 17364 raft_consensus.cc:729] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8614bbdff011473abb0f0cfce354d4a6, State: Initialized, Role: LEARNER
I20240503 14:00:07.766463 17364 consensus_queue.cc:260] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: true } }
I20240503 14:00:07.766686 17364 ts_tablet_manager.cc:1398] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:00:07.767261 16017 tablet_copy_service.cc:342] P e90a2a2980f84d49a2532f143f71e753: Request end of tablet copy session 8614bbdff011473abb0f0cfce354d4a6-274cf23ceaaa4a049f8c35689918d7b4 received from {username='slave'} at 127.0.99.131:58951
I20240503 14:00:07.767347 16017 tablet_copy_service.cc:433] P e90a2a2980f84d49a2532f143f71e753: ending tablet copy session 8614bbdff011473abb0f0cfce354d4a6-274cf23ceaaa4a049f8c35689918d7b4 on tablet 274cf23ceaaa4a049f8c35689918d7b4 with peer 8614bbdff011473abb0f0cfce354d4a6
I20240503 14:00:07.793522 17416 ts_tablet_manager.cc:902] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: Initiating tablet copy from peer 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623)
I20240503 14:00:07.794080 17416 tablet_copy_client.cc:308] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:44623
I20240503 14:00:07.794688 15008 tablet_copy_service.cc:140] P 89314f1a84d8474dba0db8615c043d1a: Received BeginTabletCopySession request for tablet ce6f858488e54943b56317d074f939e6 from peer e90a2a2980f84d49a2532f143f71e753 ({username='slave'} at 127.0.99.134:48427)
I20240503 14:00:07.794855 15008 tablet_copy_service.cc:161] P 89314f1a84d8474dba0db8615c043d1a: Beginning new tablet copy session on tablet ce6f858488e54943b56317d074f939e6 from peer e90a2a2980f84d49a2532f143f71e753 at {username='slave'} at 127.0.99.134:48427: session id = e90a2a2980f84d49a2532f143f71e753-ce6f858488e54943b56317d074f939e6
I20240503 14:00:07.795619 15008 tablet_copy_source_session.cc:205] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.796685 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f858488e54943b56317d074f939e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.798640 17416 tablet_copy_client.cc:783] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.798779 17416 tablet_copy_client.cc:647] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.799960 17416 tablet_copy_client.cc:523] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.801134 17416 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: Bootstrap starting.
I20240503 14:00:07.816197 17422 ts_tablet_manager.cc:902] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: Initiating tablet copy from peer 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449)
I20240503 14:00:07.816790 17422 tablet_copy_client.cc:308] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43449
I20240503 14:00:07.817423 14876 tablet_copy_service.cc:140] P 6ecce61b6a854f02a65cfabb70a738ea: Received BeginTabletCopySession request for tablet bc5e78a9d4b94f79be8c01753d403f58 from peer e90a2a2980f84d49a2532f143f71e753 ({username='slave'} at 127.0.99.134:45331)
I20240503 14:00:07.817548 14876 tablet_copy_service.cc:161] P 6ecce61b6a854f02a65cfabb70a738ea: Beginning new tablet copy session on tablet bc5e78a9d4b94f79be8c01753d403f58 from peer e90a2a2980f84d49a2532f143f71e753 at {username='slave'} at 127.0.99.134:45331: session id = e90a2a2980f84d49a2532f143f71e753-bc5e78a9d4b94f79be8c01753d403f58
I20240503 14:00:07.818531 14876 tablet_copy_source_session.cc:205] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.819550 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5e78a9d4b94f79be8c01753d403f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.821195 17416 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.821429 17422 tablet_copy_client.cc:783] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.821447 17416 tablet_bootstrap.cc:492] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: Bootstrap complete.
I20240503 14:00:07.821611 17422 tablet_copy_client.cc:647] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.821692 17416 ts_tablet_manager.cc:1367] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: Time spent bootstrapping tablet: real 0.021s user 0.021s sys 0.000s
I20240503 14:00:07.822183 17416 raft_consensus.cc:348] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } }
I20240503 14:00:07.822324 17416 raft_consensus.cc:729] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Initialized, Role: LEARNER
I20240503 14:00:07.822434 17416 consensus_queue.cc:260] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: true } }
I20240503 14:00:07.822806 17416 ts_tablet_manager.cc:1398] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:07.823132 17422 tablet_copy_client.cc:523] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.823534 15008 tablet_copy_service.cc:342] P 89314f1a84d8474dba0db8615c043d1a: Request end of tablet copy session e90a2a2980f84d49a2532f143f71e753-ce6f858488e54943b56317d074f939e6 received from {username='slave'} at 127.0.99.134:48427
I20240503 14:00:07.823612 15008 tablet_copy_service.cc:433] P 89314f1a84d8474dba0db8615c043d1a: ending tablet copy session e90a2a2980f84d49a2532f143f71e753-ce6f858488e54943b56317d074f939e6 on tablet ce6f858488e54943b56317d074f939e6 with peer e90a2a2980f84d49a2532f143f71e753
I20240503 14:00:07.824581 17422 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: Bootstrap starting.
I20240503 14:00:07.828641 17428 ts_tablet_manager.cc:902] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Initiating tablet copy from peer 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449)
I20240503 14:00:07.829344 17428 tablet_copy_client.cc:308] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43449
I20240503 14:00:07.830025 14876 tablet_copy_service.cc:140] P 6ecce61b6a854f02a65cfabb70a738ea: Received BeginTabletCopySession request for tablet bc5e78a9d4b94f79be8c01753d403f58 from peer 89314f1a84d8474dba0db8615c043d1a ({username='slave'} at 127.0.99.130:42397)
I20240503 14:00:07.830179 14876 tablet_copy_service.cc:161] P 6ecce61b6a854f02a65cfabb70a738ea: Beginning new tablet copy session on tablet bc5e78a9d4b94f79be8c01753d403f58 from peer 89314f1a84d8474dba0db8615c043d1a at {username='slave'} at 127.0.99.130:42397: session id = 89314f1a84d8474dba0db8615c043d1a-bc5e78a9d4b94f79be8c01753d403f58
I20240503 14:00:07.831046 14876 tablet_copy_source_session.cc:205] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:07.832111 17428 ts_tablet_manager.cc:1875] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:07.834309 17428 ts_tablet_manager.cc:1888] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:00:07.834504 17428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5e78a9d4b94f79be8c01753d403f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.835649 17428 tablet_copy_client.cc:783] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:07.835835 17428 tablet_copy_client.cc:647] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:07.837320 17428 tablet_copy_client.cc:523] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:07.838917 17428 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap starting.
I20240503 14:00:07.845052 17422 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.845338 17422 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: Bootstrap complete.
I20240503 14:00:07.845525 17422 ts_tablet_manager.cc:1367] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20240503 14:00:07.845952 17422 raft_consensus.cc:348] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } }
I20240503 14:00:07.846117 17422 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Initialized, Role: LEARNER
I20240503 14:00:07.846227 17422 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } }
I20240503 14:00:07.846539 17422 ts_tablet_manager.cc:1398] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:00:07.847261 14876 tablet_copy_service.cc:342] P 6ecce61b6a854f02a65cfabb70a738ea: Request end of tablet copy session e90a2a2980f84d49a2532f143f71e753-bc5e78a9d4b94f79be8c01753d403f58 received from {username='slave'} at 127.0.99.134:45331
I20240503 14:00:07.847353 14876 tablet_copy_service.cc:433] P 6ecce61b6a854f02a65cfabb70a738ea: ending tablet copy session e90a2a2980f84d49a2532f143f71e753-bc5e78a9d4b94f79be8c01753d403f58 on tablet bc5e78a9d4b94f79be8c01753d403f58 with peer e90a2a2980f84d49a2532f143f71e753
I20240503 14:00:07.859645 17428 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:07.859988 17428 tablet_bootstrap.cc:492] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Bootstrap complete.
I20240503 14:00:07.860184 17428 ts_tablet_manager.cc:1367] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.003s
I20240503 14:00:07.860631 17428 raft_consensus.cc:348] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } }
I20240503 14:00:07.860755 17428 raft_consensus.cc:729] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 89314f1a84d8474dba0db8615c043d1a, State: Initialized, Role: LEARNER
I20240503 14:00:07.860846 17428 consensus_queue.cc:260] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } }
I20240503 14:00:07.861079 17428 ts_tablet_manager.cc:1398] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:07.861825 14876 tablet_copy_service.cc:342] P 6ecce61b6a854f02a65cfabb70a738ea: Request end of tablet copy session 89314f1a84d8474dba0db8615c043d1a-bc5e78a9d4b94f79be8c01753d403f58 received from {username='slave'} at 127.0.99.130:42397
I20240503 14:00:07.861937 14876 tablet_copy_service.cc:433] P 6ecce61b6a854f02a65cfabb70a738ea: ending tablet copy session 89314f1a84d8474dba0db8615c043d1a-bc5e78a9d4b94f79be8c01753d403f58 on tablet bc5e78a9d4b94f79be8c01753d403f58 with peer 89314f1a84d8474dba0db8615c043d1a
I20240503 14:00:08.158557 15120 raft_consensus.cc:1203] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:00:08.202173 14855 raft_consensus.cc:1203] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:08.204643 15996 raft_consensus.cc:1203] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 14:00:08.233942 15866 raft_consensus.cc:1203] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:00:08.265242 15120 raft_consensus.cc:1203] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:00:08.340662 14988 raft_consensus.cc:1203] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 14:00:08.385393 15996 raft_consensus.cc:1203] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:08.602672 17508 raft_consensus.cc:1050] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: attempting to promote NON_VOTER e90a2a2980f84d49a2532f143f71e753 to VOTER
I20240503 14:00:08.603194 17508 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } }
I20240503 14:00:08.604924 15996 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEARNER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.604962 15120 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.605082 16128 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.605418 17507 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.605726 17507 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.606050 14988 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEARNER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.606340 17507 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.607420 17507 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.608104 17507 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:08.610222 15996 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:08.610404 17507 raft_consensus.cc:1050] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea: attempting to promote NON_VOTER 89314f1a84d8474dba0db8615c043d1a to VOTER
I20240503 14:00:08.610730 17507 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } }
I20240503 14:00:08.610702 15120 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:08.610756 14711 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:08.611945 14988 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:08.615423 15120 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:08.615423 15996 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:08.615665 14988 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEARNER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:08.615657 16128 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: true } } }
I20240503 14:00:08.616017 17567 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:08.616139 17508 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:08.616227 17568 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:08.620049 17567 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.620831 15120 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.622292 15996 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.622972 14711 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 10 to 11, 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:08.625205 16128 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.625277 14988 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.626776 14855 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } }
I20240503 14:00:08.628352 15996 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:00:08.628855 17567 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:08.629321 14988 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:00:08.629796 17567 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:08.631243 16127 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:00:08.632333 17507 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:08.632565 17568 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.633062 15996 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.635468 14714 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 11 to 12, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:08.635736 16127 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.636991 14988 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.637787 14698 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc5e78a9d4b94f79be8c01753d403f58 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:08.641520 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5e78a9d4b94f79be8c01753d403f58 with delete_type TABLET_DATA_TOMBSTONED (TS 8614bbdff011473abb0f0cfce354d4a6 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37684
I20240503 14:00:08.641914 17593 tablet_replica.cc:330] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: stopping tablet replica
I20240503 14:00:08.642148 17593 raft_consensus.cc:2229] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:08.642427 17593 raft_consensus.cc:2258] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:08.643177 14855 consensus_queue.cc:237] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } }
I20240503 14:00:08.644402 15996 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:00:08.644444 14988 raft_consensus.cc:1261] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:00:08.644603 17593 ts_tablet_manager.cc:1875] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:08.644860 17507 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:00:08.645036 17567 consensus_queue.cc:1035] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:00:08.646934 17567 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.647526 15996 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.648329 17593 ts_tablet_manager.cc:1888] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20240503 14:00:08.648411 17593 log.cc:1196] T bc5e78a9d4b94f79be8c01753d403f58 P 8614bbdff011473abb0f0cfce354d4a6: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc5e78a9d4b94f79be8c01753d403f58
I20240503 14:00:08.650183 14714 catalog_manager.cc:5484] T bc5e78a9d4b94f79be8c01753d403f58 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 12 to 13, VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:08.650373 14698 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc5e78a9d4b94f79be8c01753d403f58 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:08.657565 16108 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5e78a9d4b94f79be8c01753d403f58 with delete_type TABLET_DATA_TOMBSTONED (TS fb0b622a674141069043d816ad306f3e not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:58382
I20240503 14:00:08.658138 17596 tablet_replica.cc:330] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: stopping tablet replica
I20240503 14:00:08.658520 14700 catalog_manager.cc:4830] TS 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): tablet bc5e78a9d4b94f79be8c01753d403f58 (table rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]) successfully deleted
I20240503 14:00:08.659847 17596 raft_consensus.cc:2229] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:08.660292 17596 raft_consensus.cc:2258] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:08.661347 14988 raft_consensus.cc:2932] T bc5e78a9d4b94f79be8c01753d403f58 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { promote: false } } }
I20240503 14:00:08.661864 17596 ts_tablet_manager.cc:1875] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:08.664573 17596 ts_tablet_manager.cc:1888] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 14:00:08.664759 17596 log.cc:1196] T bc5e78a9d4b94f79be8c01753d403f58 P fb0b622a674141069043d816ad306f3e: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/bc5e78a9d4b94f79be8c01753d403f58
I20240503 14:00:08.665377 14700 catalog_manager.cc:4830] TS fb0b622a674141069043d816ad306f3e (127.0.99.133:46269): tablet bc5e78a9d4b94f79be8c01753d403f58 (table rebalance_test_table_2 [id=b8993f498d0a42e4b205ce7ba012d74f]) successfully deleted
I20240503 14:00:08.681344 17505 raft_consensus.cc:1050] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER 6ecce61b6a854f02a65cfabb70a738ea to VOTER
I20240503 14:00:08.681859 17505 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } }
I20240503 14:00:08.683684 15866 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:08.683722 15996 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:08.683722 15120 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:08.683750 14855 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:08.684201 17598 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.684417 17506 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.684548 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.684705 17505 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.687033 17598 raft_consensus.cc:1013] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: attempt to promote peer e90a2a2980f84d49a2532f143f71e753: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 14:00:08.687146 17598 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.687659 15120 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.688014 15866 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.689940 15996 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.690438 17086 raft_consensus.cc:1050] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER 35b5bdde0c49481e8d69060ad4d1d81d to VOTER
I20240503 14:00:08.690420 14714 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 8 to 9, 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.690801 17086 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:08.692755 15996 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.693037 17086 raft_consensus.cc:1050] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a: attempting to promote NON_VOTER e90a2a2980f84d49a2532f143f71e753 to VOTER
I20240503 14:00:08.693231 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.693401 15866 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.693418 17086 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } }
I20240503 14:00:08.693646 14855 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: true } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.694401 17086 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.694574 17605 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.695377 15866 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:08.695432 15996 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 LEARNER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:08.696659 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.697005 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.699051 17606 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.699087 17086 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.700285 14856 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
W20240503 14:00:08.701150 14700 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet ce6f858488e54943b56317d074f939e6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 9 but the committed config has opid_index of 10
I20240503 14:00:08.701064 15997 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:08.701049 15865 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.701536 14855 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:08.702044 17506 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.702983 15996 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.703557 17605 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:08.703740 14713 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d reported cstate change: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.704852 14711 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 8 to 9, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:08.705770 15865 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:08.709666 14855 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.710350 15120 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:08.711632 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.712949 15120 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { replace: true } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:08.720067 17063 raft_consensus.cc:1050] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753: attempting to promote NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 to VOTER
I20240503 14:00:08.720528 17063 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } }
I20240503 14:00:08.722162 15120 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.722157 14856 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.722158 14988 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.722602 17611 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.722841 17063 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.723011 17494 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.725298 17611 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.725642 14988 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.725730 14856 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.728793 15120 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.728931 14714 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { replace: true } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.732461 15996 consensus_queue.cc:237] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } }
I20240503 14:00:08.733865 15120 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.733922 14988 raft_consensus.cc:1261] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.734977 17494 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.735188 17063 consensus_queue.cc:1035] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.736930 17510 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.737771 15120 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.738854 14701 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 274cf23ceaaa4a049f8c35689918d7b4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:08.739431 14988 raft_consensus.cc:2932] T 274cf23ceaaa4a049f8c35689918d7b4 P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.740589 14713 catalog_manager.cc:5484] T 274cf23ceaaa4a049f8c35689918d7b4 P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 9 to 10, VOTER 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.743518 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet 274cf23ceaaa4a049f8c35689918d7b4 with delete_type TABLET_DATA_TOMBSTONED (TS 6ecce61b6a854f02a65cfabb70a738ea not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40884
I20240503 14:00:08.744140 17615 tablet_replica.cc:330] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: stopping tablet replica
I20240503 14:00:08.744287 17615 raft_consensus.cc:2229] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:08.744778 17615 raft_consensus.cc:2258] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:08.746071 17615 ts_tablet_manager.cc:1875] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:08.748669 17615 ts_tablet_manager.cc:1888] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:00:08.748766 17615 log.cc:1196] T 274cf23ceaaa4a049f8c35689918d7b4 P 6ecce61b6a854f02a65cfabb70a738ea: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/274cf23ceaaa4a049f8c35689918d7b4
I20240503 14:00:08.749414 14698 catalog_manager.cc:4830] TS 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129:43449): tablet 274cf23ceaaa4a049f8c35689918d7b4 (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:08.765942 17494 raft_consensus.cc:1050] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: attempting to promote NON_VOTER 8614bbdff011473abb0f0cfce354d4a6 to VOTER
I20240503 14:00:08.766458 17494 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } }
I20240503 14:00:08.768034 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.768059 15120 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 LEARNER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.768087 14988 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:08.768427 17611 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.768590 17494 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.768779 17063 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:08.770888 17611 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.771301 14988 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.771756 15120 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.773808 14711 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 reported cstate change: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:08.775365 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } attrs { replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.780076 15996 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } }
I20240503 14:00:08.781411 15120 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.781682 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:08.781862 17494 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.782076 17611 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:08.784189 17494 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.784509 15120 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.784509 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.786352 14701 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d68b8332ce64eb2935deba765c3946a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:08.786899 14712 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 9 to 10, VOTER 89314f1a84d8474dba0db8615c043d1a (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } }
I20240503 14:00:08.790071 14968 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d68b8332ce64eb2935deba765c3946a with delete_type TABLET_DATA_TOMBSTONED (TS 89314f1a84d8474dba0db8615c043d1a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50526
I20240503 14:00:08.790871 17635 tablet_replica.cc:330] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: stopping tablet replica
I20240503 14:00:08.791076 17635 raft_consensus.cc:2229] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:08.791312 17635 raft_consensus.cc:2258] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:08.792141 17635 ts_tablet_manager.cc:1875] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:08.794291 17635 ts_tablet_manager.cc:1888] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:00:08.794384 17635 log.cc:1196] T 6d68b8332ce64eb2935deba765c3946a P 89314f1a84d8474dba0db8615c043d1a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d68b8332ce64eb2935deba765c3946a
I20240503 14:00:08.794929 14700 catalog_manager.cc:4830] TS 89314f1a84d8474dba0db8615c043d1a (127.0.99.130:44623): tablet 6d68b8332ce64eb2935deba765c3946a (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:09.135066 14988 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } }
I20240503 14:00:09.136484 15996 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:00:09.136495 15865 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:00:09.136572 14856 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:00:09.136916 17606 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:09.137094 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:09.137233 17599 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:09.139058 17606 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.139537 15865 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.139689 15996 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.141048 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce6f858488e54943b56317d074f939e6 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:09.141449 14856 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.142702 14712 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 reported cstate change: config changed from index 10 to 11, VOTER 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false replace: true } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.145262 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6f858488e54943b56317d074f939e6 with delete_type TABLET_DATA_TOMBSTONED (TS 8614bbdff011473abb0f0cfce354d4a6 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:37684
I20240503 14:00:09.145521 17593 tablet_replica.cc:330] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: stopping tablet replica
I20240503 14:00:09.145678 17593 raft_consensus.cc:2229] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:09.145893 17593 raft_consensus.cc:2258] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:09.145907 14988 consensus_queue.cc:237] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } }
I20240503 14:00:09.146809 17593 ts_tablet_manager.cc:1875] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:09.147308 14856 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:00:09.147665 15996 raft_consensus.cc:1261] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:00:09.147701 17086 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:09.148048 17599 consensus_queue.cc:1035] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:09.149204 17593 ts_tablet_manager.cc:1888] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 14:00:09.149262 17593 log.cc:1196] T ce6f858488e54943b56317d074f939e6 P 8614bbdff011473abb0f0cfce354d4a6: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ce6f858488e54943b56317d074f939e6
I20240503 14:00:09.149760 17086 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.149921 14700 catalog_manager.cc:4830] TS 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503): tablet ce6f858488e54943b56317d074f939e6 (table rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]) successfully deleted
I20240503 14:00:09.150254 15996 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P e90a2a2980f84d49a2532f143f71e753 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.151494 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce6f858488e54943b56317d074f939e6 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:09.151861 14856 raft_consensus.cc:2932] T ce6f858488e54943b56317d074f939e6 P 6ecce61b6a854f02a65cfabb70a738ea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } } }
I20240503 14:00:09.152312 14714 catalog_manager.cc:5484] T ce6f858488e54943b56317d074f939e6 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 11 to 12, VOTER 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:09.158607 15844 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6f858488e54943b56317d074f939e6 with delete_type TABLET_DATA_TOMBSTONED (TS 35b5bdde0c49481e8d69060ad4d1d81d not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:57906
I20240503 14:00:09.159173 17687 tablet_replica.cc:330] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: stopping tablet replica
I20240503 14:00:09.159484 17687 raft_consensus.cc:2229] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:09.159795 17687 raft_consensus.cc:2258] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:09.160794 17687 ts_tablet_manager.cc:1875] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:09.162778 17687 ts_tablet_manager.cc:1888] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 14:00:09.162876 17687 log.cc:1196] T ce6f858488e54943b56317d074f939e6 P 35b5bdde0c49481e8d69060ad4d1d81d: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ce6f858488e54943b56317d074f939e6
I20240503 14:00:09.163520 14701 catalog_manager.cc:4830] TS 35b5bdde0c49481e8d69060ad4d1d81d (127.0.99.132:32851): tablet ce6f858488e54943b56317d074f939e6 (table rebalance_test_table_1 [id=acef8c69f8004c9c9d72384128664102]) successfully deleted
I20240503 14:00:09.243145 14988 consensus_queue.cc:237] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } }
I20240503 14:00:09.244441 15865 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:09.244832 14856 raft_consensus.cc:1261] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer 89314f1a84d8474dba0db8615c043d1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:09.244892 17605 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:09.245206 17505 consensus_queue.cc:1035] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:09.246704 17605 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:09.247265 14856 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:09.248744 15865 raft_consensus.cc:2932] T eb5245fe54de4741ada591ab9576e699 P 35b5bdde0c49481e8d69060ad4d1d81d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } } }
I20240503 14:00:09.249155 14711 catalog_manager.cc:5484] T eb5245fe54de4741ada591ab9576e699 P 89314f1a84d8474dba0db8615c043d1a reported cstate change: config changed from index 9 to 10, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "89314f1a84d8474dba0db8615c043d1a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89314f1a84d8474dba0db8615c043d1a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b5bdde0c49481e8d69060ad4d1d81d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 32851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:09.250275 14700 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb5245fe54de4741ada591ab9576e699 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:09.252038 15970 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5245fe54de4741ada591ab9576e699 with delete_type TABLET_DATA_TOMBSTONED (TS e90a2a2980f84d49a2532f143f71e753 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47932
I20240503 14:00:09.252750 17703 tablet_replica.cc:330] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: stopping tablet replica
I20240503 14:00:09.253049 17703 raft_consensus.cc:2229] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:09.253314 17703 raft_consensus.cc:2258] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:09.254308 17703 ts_tablet_manager.cc:1875] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:09.256170 17703 ts_tablet_manager.cc:1888] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:00:09.256253 17703 log.cc:1196] T eb5245fe54de4741ada591ab9576e699 P e90a2a2980f84d49a2532f143f71e753: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb5245fe54de4741ada591ab9576e699
I20240503 14:00:09.256819 14701 catalog_manager.cc:4830] TS e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): tablet eb5245fe54de4741ada591ab9576e699 (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:09.341547 14836 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:09.342779 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:09.346027 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:09.347437 14968 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:09.347748 15098 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:09.368984 15970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:09.499792 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:09.502933 14836 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:09.505149 15098 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:09.505928 14968 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:09.506373 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:09.507278 15970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:09.550506 15996 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:09.552201 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:09.552443 15120 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:09.552696 17780 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:09.552912 17779 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:09.554920 17510 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:09.555392 15120 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:09.555437 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
I20240503 14:00:09.557734 14714 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 10 to 11, NON_VOTER fb0b622a674141069043d816ad306f3e (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } } }
W20240503 14:00:09.559881 15932 consensus_peers.cc:485] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 -> Peer fb0b622a674141069043d816ad306f3e (127.0.99.133:46269): Couldn't send request to peer fb0b622a674141069043d816ad306f3e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d68b8332ce64eb2935deba765c3946a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:09.967437 17792 ts_tablet_manager.cc:902] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: Initiating tablet copy from peer e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635)
I20240503 14:00:09.967927 17792 tablet_copy_client.cc:308] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:38635
I20240503 14:00:09.971696 16017 tablet_copy_service.cc:140] P e90a2a2980f84d49a2532f143f71e753: Received BeginTabletCopySession request for tablet 6d68b8332ce64eb2935deba765c3946a from peer fb0b622a674141069043d816ad306f3e ({username='slave'} at 127.0.99.133:54159)
I20240503 14:00:09.971838 16017 tablet_copy_service.cc:161] P e90a2a2980f84d49a2532f143f71e753: Beginning new tablet copy session on tablet 6d68b8332ce64eb2935deba765c3946a from peer fb0b622a674141069043d816ad306f3e at {username='slave'} at 127.0.99.133:54159: session id = fb0b622a674141069043d816ad306f3e-6d68b8332ce64eb2935deba765c3946a
I20240503 14:00:09.972723 16017 tablet_copy_source_session.cc:205] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:09.973742 17792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d68b8332ce64eb2935deba765c3946a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.976064 17792 tablet_copy_client.cc:783] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:09.976243 17792 tablet_copy_client.cc:647] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:09.977597 17792 tablet_copy_client.cc:523] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:09.978993 17792 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: Bootstrap starting.
I20240503 14:00:09.999811 17792 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:10.000097 17792 tablet_bootstrap.cc:492] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: Bootstrap complete.
I20240503 14:00:10.000288 17792 ts_tablet_manager.cc:1367] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240503 14:00:10.000742 17792 raft_consensus.cc:348] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:10.000869 17792 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fb0b622a674141069043d816ad306f3e, State: Initialized, Role: LEARNER
I20240503 14:00:10.000967 17792 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: true } }
I20240503 14:00:10.001339 17792 ts_tablet_manager.cc:1398] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:00:10.002075 16017 tablet_copy_service.cc:342] P e90a2a2980f84d49a2532f143f71e753: Request end of tablet copy session fb0b622a674141069043d816ad306f3e-6d68b8332ce64eb2935deba765c3946a received from {username='slave'} at 127.0.99.133:54159
I20240503 14:00:10.002177 16017 tablet_copy_service.cc:433] P e90a2a2980f84d49a2532f143f71e753: ending tablet copy session fb0b622a674141069043d816ad306f3e-6d68b8332ce64eb2935deba765c3946a on tablet 6d68b8332ce64eb2935deba765c3946a with peer fb0b622a674141069043d816ad306f3e
I20240503 14:00:10.566469 16127 raft_consensus.cc:1203] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240503 14:00:11.093806 17816 raft_consensus.cc:1050] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: attempting to promote NON_VOTER fb0b622a674141069043d816ad306f3e to VOTER
I20240503 14:00:11.094314 17816 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:11.095883 16127 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 2 LEARNER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:00:11.095937 14856 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:00:11.096127 15120 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Refusing update from remote peer e90a2a2980f84d49a2532f143f71e753: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:00:11.096341 17836 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:11.096581 17816 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:11.096802 17833 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:11.099136 17816 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:11.099731 16127 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:11.100675 15120 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:11.101351 14856 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:11.103725 14713 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 reported cstate change: config changed from index 11 to 12, fb0b622a674141069043d816ad306f3e (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e90a2a2980f84d49a2532f143f71e753" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:11.225764 15996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "e90a2a2980f84d49a2532f143f71e753"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46718
I20240503 14:00:11.226022 15996 raft_consensus.cc:595] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:00:11.438563 15996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "e90a2a2980f84d49a2532f143f71e753"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46732
I20240503 14:00:11.438735 15996 raft_consensus.cc:595] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:00:11.515594 17836 raft_consensus.cc:979] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: : Instructing follower 6ecce61b6a854f02a65cfabb70a738ea to start an election
I20240503 14:00:11.515718 17836 raft_consensus.cc:1067] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Signalling peer 6ecce61b6a854f02a65cfabb70a738ea to start an election
I20240503 14:00:11.516294 14856 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a"
dest_uuid: "6ecce61b6a854f02a65cfabb70a738ea"
from {username='slave'} at 127.0.99.134:45331
I20240503 14:00:11.516445 14856 raft_consensus.cc:482] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:11.516505 14856 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:11.517419 14856 raft_consensus.cc:504] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:11.517881 14856 leader_election.cc:288] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 3 election: Requested vote from peers e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635), 8614bbdff011473abb0f0cfce354d4a6 (127.0.99.131:45503), fb0b622a674141069043d816ad306f3e (127.0.99.133:46269)
I20240503 14:00:11.518319 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "8614bbdff011473abb0f0cfce354d4a6"
I20240503 14:00:11.518474 15120 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:11.518458 15996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "e90a2a2980f84d49a2532f143f71e753"
I20240503 14:00:11.518496 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d68b8332ce64eb2935deba765c3946a" candidate_uuid: "6ecce61b6a854f02a65cfabb70a738ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "fb0b622a674141069043d816ad306f3e"
I20240503 14:00:11.518591 15996 raft_consensus.cc:3032] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:00:11.518616 16127 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:11.518638 15996 raft_consensus.cc:729] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e90a2a2980f84d49a2532f143f71e753, State: Running, Role: LEADER
I20240503 14:00:11.518826 15996 consensus_queue.cc:260] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:11.519032 15996 raft_consensus.cc:3037] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:11.519559 16127 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 3.
I20240503 14:00:11.519559 15120 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 3.
I20240503 14:00:11.519920 15996 raft_consensus.cc:2454] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecce61b6a854f02a65cfabb70a738ea in term 3.
I20240503 14:00:11.519989 14791 leader_election.cc:302] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecce61b6a854f02a65cfabb70a738ea, 8614bbdff011473abb0f0cfce354d4a6, fb0b622a674141069043d816ad306f3e; no voters:
I20240503 14:00:11.520207 17508 raft_consensus.cc:2781] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:00:11.520385 17508 raft_consensus.cc:686] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 3 LEADER]: Becoming Leader. State: Replica: 6ecce61b6a854f02a65cfabb70a738ea, State: Running, Role: LEADER
I20240503 14:00:11.520560 17508 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:11.522382 14714 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: term changed from 2 to 3, leader changed from e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) to 6ecce61b6a854f02a65cfabb70a738ea (127.0.99.129). New cstate: current_term: 3 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:11.900094 15996 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 3 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 14:00:11.900627 17508 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90a2a2980f84d49a2532f143f71e753" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38635 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:00:12.039971 16127 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 3 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 14:00:12.040484 17508 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:00:12.043889 15120 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 3 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 14:00:12.044411 17878 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:00:12.596889 14856 consensus_queue.cc:237] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } }
I20240503 14:00:12.598228 16127 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 3 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 14:00:12.598263 15120 raft_consensus.cc:1261] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 3 FOLLOWER]: Refusing update from remote peer 6ecce61b6a854f02a65cfabb70a738ea: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 14:00:12.598644 17901 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 14:00:12.598891 17900 consensus_queue.cc:1035] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 14:00:12.600752 17880 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:12.601284 16127 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P fb0b622a674141069043d816ad306f3e [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:12.601312 15120 raft_consensus.cc:2932] T 6d68b8332ce64eb2935deba765c3946a P 8614bbdff011473abb0f0cfce354d4a6 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } } }
I20240503 14:00:12.602785 14698 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d68b8332ce64eb2935deba765c3946a with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:12.603590 14712 catalog_manager.cc:5484] T 6d68b8332ce64eb2935deba765c3946a P 6ecce61b6a854f02a65cfabb70a738ea reported cstate change: config changed from index 12 to 14, VOTER e90a2a2980f84d49a2532f143f71e753 (127.0.99.134) evicted. New cstate: current_term: 3 leader_uuid: "6ecce61b6a854f02a65cfabb70a738ea" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecce61b6a854f02a65cfabb70a738ea" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614bbdff011473abb0f0cfce354d4a6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0b622a674141069043d816ad306f3e" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46269 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:12.606814 15970 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d68b8332ce64eb2935deba765c3946a with delete_type TABLET_DATA_TOMBSTONED (TS e90a2a2980f84d49a2532f143f71e753 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:47932
I20240503 14:00:12.607213 17907 tablet_replica.cc:330] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: stopping tablet replica
I20240503 14:00:12.607358 17907 raft_consensus.cc:2229] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:12.607523 17907 raft_consensus.cc:2258] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:12.608246 17907 ts_tablet_manager.cc:1875] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:12.610421 17907 ts_tablet_manager.cc:1888] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240503 14:00:12.610502 17907 log.cc:1196] T 6d68b8332ce64eb2935deba765c3946a P e90a2a2980f84d49a2532f143f71e753: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714744724484359-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/6d68b8332ce64eb2935deba765c3946a
I20240503 14:00:12.611059 14701 catalog_manager.cc:4830] TS e90a2a2980f84d49a2532f143f71e753 (127.0.99.134:38635): tablet 6d68b8332ce64eb2935deba765c3946a (table rebalance_test_table_0 [id=f987d80302da44f5a7a26bde4849bbc2]) successfully deleted
I20240503 14:00:12.747756 14968 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:12.749555 15098 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:12.749568 14836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:12.752743 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:12.772015 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:12.776718 15970 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:12.855540 15844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:12.860548 14836 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:12.861478 15098 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:12.868291 16108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:12.868350 14968 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:12.880841 15970 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:12.916832 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 14774
I20240503 14:00:12.924208 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 14905
I20240503 14:00:12.930601 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 15037
I20240503 14:00:12.937559 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 15784
I20240503 14:00:12.943706 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 16046
I20240503 14:00:12.949122 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 15915
I20240503 14:00:12.955090 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 14683
2024-05-03T14:00:12Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (17166 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240503 14:00:12.982864 398 test_util.cc:268] Using random seed: 594891335
I20240503 14:00:12.983785 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:00:12.983832 398 ts_itest-base.cc:113] --------------
I20240503 14:00:12.983847 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:00:12.983860 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:00:12.983871 398 ts_itest-base.cc:116] --------------
2024-05-03T14:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:00:12Z Disabled control of system clock
I20240503 14:00:12.995074 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35671
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35671
--location_mapping_cmd=/tmp/dist-test-taskmJWZN5/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 14:00:13.115046 17982 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:00:13.115386 17982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.115424 17982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.115490 17982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.118896 17982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:13.118996 17982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:00:13.119024 17982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.119040 17982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:00:13.119056 17982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:00:13.123270 17982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskmJWZN5/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35671
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.124415 17982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.125823 17982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.135454 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.136384 17982 server_base.cc:1034] running on GCE node
W20240503 14:00:13.137228 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.138311 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.139000 17982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.140185 17982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.141379 17982 hybrid_clock.cc:648] HybridClock initialized: now 1714744813141346 us; error 50 us; skew 500 ppm
I20240503 14:00:13.143869 17982 webserver.cc:460] Webserver started at http://127.0.99.190:43117/ using document root <none> and password file <none>
I20240503 14:00:13.144552 17982 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.144623 17982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.144827 17982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:13.146837 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8324f35c19624f00854c9ab58b6383c5"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.147409 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8324f35c19624f00854c9ab58b6383c5"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.151578 17982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:00:13.154675 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.156196 17982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:00:13.156396 17982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8324f35c19624f00854c9ab58b6383c5"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.156523 17982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.184587 17982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.185408 17982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.185604 17982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.192831 17982 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35671
I20240503 14:00:13.192831 18048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35671 every 8 connection(s)
I20240503 14:00:13.194003 17982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:00:13.197001 18049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:13.202121 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 17982
I20240503 14:00:13.202253 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:00:13.203226 18049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5: Bootstrap starting.
I20240503 14:00:13.205744 18049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:13.206722 18049 log.cc:824] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:13.208976 18049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5: No bootstrap required, opened a new log
I20240503 14:00:13.212481 18049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8324f35c19624f00854c9ab58b6383c5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35671 } }
I20240503 14:00:13.212786 18049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:13.212869 18049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324f35c19624f00854c9ab58b6383c5, State: Initialized, Role: FOLLOWER
I20240503 14:00:13.213382 18049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8324f35c19624f00854c9ab58b6383c5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35671 } }
I20240503 14:00:13.213555 18049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:00:13.213689 18049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:00:13.213809 18049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:13.214722 18049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8324f35c19624f00854c9ab58b6383c5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35671 } }
I20240503 14:00:13.215076 18049 leader_election.cc:302] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8324f35c19624f00854c9ab58b6383c5; no voters:
I20240503 14:00:13.215377 18049 leader_election.cc:288] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:00:13.215828 18054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:13.216040 18054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8324f35c19624f00854c9ab58b6383c5, State: Running, Role: LEADER
I20240503 14:00:13.216359 18054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8324f35c19624f00854c9ab58b6383c5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35671 } }
I20240503 14:00:13.216606 18049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:00:13.218108 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8324f35c19624f00854c9ab58b6383c5. Latest consensus state: current_term: 1 leader_uuid: "8324f35c19624f00854c9ab58b6383c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8324f35c19624f00854c9ab58b6383c5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35671 } } }
I20240503 14:00:13.218124 18055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8324f35c19624f00854c9ab58b6383c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8324f35c19624f00854c9ab58b6383c5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35671 } } }
I20240503 14:00:13.218287 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:13.218287 18055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5 [sys.catalog]: This master's current role is: LEADER
W20240503 14:00:13.222255 18069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 14:00:13.222352 18069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 14:00:13.222435 18066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:00:13.223229 18066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:00:13.229116 18066 catalog_manager.cc:1347] Generated new cluster ID: 358c0897856b4a589329222f4099e1ed
I20240503 14:00:13.229212 18066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:00:13.245875 18066 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:00:13.247073 18066 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:00:13.254081 18066 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8324f35c19624f00854c9ab58b6383c5: Generated new TSK 0
I20240503 14:00:13.254935 18066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:00:13.260375 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:00:13.385316 18073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.385592 18073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.385644 18073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.389123 18073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:13.389226 18073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.389300 18073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 14:00:13.393347 18073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.18073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.394457 18073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.395713 18073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.405666 18078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.406246 18079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.407228 18081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.408054 18073 server_base.cc:1034] running on GCE node
I20240503 14:00:13.408618 18073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.409327 18073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.410517 18073 hybrid_clock.cc:648] HybridClock initialized: now 1714744813410475 us; error 62 us; skew 500 ppm
I20240503 14:00:13.412940 18073 webserver.cc:460] Webserver started at http://127.0.99.129:41563/ using document root <none> and password file <none>
I20240503 14:00:13.413578 18073 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.413643 18073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.413839 18073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:13.415736 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.416292 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.420408 18073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:00:13.423130 18087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.424412 18073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:00:13.424558 18073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.424664 18073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.438138 18073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.438879 18073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.439045 18073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.439621 18073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.440629 18073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.440682 18073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.440738 18073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.440762 18073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.449610 18073 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33467
I20240503 14:00:13.449637 18200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33467 every 8 connection(s)
I20240503 14:00:13.450734 18073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:00:13.457326 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 18073
I20240503 14:00:13.457456 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:00:13.460026 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:13.460749 18201 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:13.461055 18201 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.461760 18201 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:13.499400 18013 ts_manager.cc:194] Registered new tserver with Master: a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:13.501258 18013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47107
W20240503 14:00:13.581269 18204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.581537 18204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.581585 18204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.584954 18204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:13.585047 18204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.585120 18204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 14:00:13.589211 18204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.18204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.590271 18204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.591497 18204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.601374 18213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.601886 18210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.602767 18211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.603531 18204 server_base.cc:1034] running on GCE node
I20240503 14:00:13.604106 18204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.604804 18204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.606047 18204 hybrid_clock.cc:648] HybridClock initialized: now 1714744813606015 us; error 44 us; skew 500 ppm
I20240503 14:00:13.608425 18204 webserver.cc:460] Webserver started at http://127.0.99.130:39543/ using document root <none> and password file <none>
I20240503 14:00:13.609133 18204 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.609213 18204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.609409 18204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:13.611348 18204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "aaa289c674d34661bf49dd8bd7186a15"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.611899 18204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "aaa289c674d34661bf49dd8bd7186a15"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.616025 18204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:00:13.618959 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.620138 18204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:13.620290 18204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aaa289c674d34661bf49dd8bd7186a15"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.620404 18204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.632787 18204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.633581 18204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.633801 18204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.634491 18204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.635566 18204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.635622 18204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.635681 18204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.635717 18204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.644863 18204 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36605
I20240503 14:00:13.644969 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36605 every 8 connection(s)
I20240503 14:00:13.646153 18204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:00:13.647104 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 18204
I20240503 14:00:13.647217 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:00:13.649708 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:13.657204 18333 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:13.657480 18333 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.658169 18333 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:13.710755 18013 ts_manager.cc:194] Registered new tserver with Master: aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:13.711680 18013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:47251
W20240503 14:00:13.787055 18336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.787322 18336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.787385 18336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.791055 18336 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:13.791153 18336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.791256 18336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 14:00:13.795353 18336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.18336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.796442 18336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.797859 18336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.807580 18343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.808115 18342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.809106 18345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.809823 18336 server_base.cc:1034] running on GCE node
I20240503 14:00:13.810379 18336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.811056 18336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.812233 18336 hybrid_clock.cc:648] HybridClock initialized: now 1714744813812215 us; error 35 us; skew 500 ppm
I20240503 14:00:13.814597 18336 webserver.cc:460] Webserver started at http://127.0.99.131:35993/ using document root <none> and password file <none>
I20240503 14:00:13.815222 18336 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.815287 18336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.815470 18336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:13.817499 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "625639f1ddfa446abaa3962865411504"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.818094 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "625639f1ddfa446abaa3962865411504"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.821843 18336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:00:13.824489 18351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.825700 18336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:13.825830 18336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "625639f1ddfa446abaa3962865411504"
format_stamp: "Formatted at 2024-05-03 14:00:13 on dist-test-slave-6fwv"
I20240503 14:00:13.825933 18336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.846668 18336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.847434 18336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.847604 18336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.848199 18336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.849208 18336 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.849259 18336 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.849323 18336 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.849347 18336 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.858284 18336 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:33181
I20240503 14:00:13.858299 18464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:33181 every 8 connection(s)
I20240503 14:00:13.859383 18336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:00:13.867913 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 18336
I20240503 14:00:13.868041 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:00:13.869451 18465 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:13.869746 18465 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.870393 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:13.870431 18465 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:13.910893 18013 ts_manager.cc:194] Registered new tserver with Master: 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:13.911778 18013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:51423
W20240503 14:00:13.987162 18468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.987426 18468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.987493 18468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.990830 18468 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:13.990923 18468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.991015 18468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 14:00:13.994956 18468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.18468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.996002 18468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.997227 18468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:14.007083 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:14.007743 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:14.008498 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:14.009403 18468 server_base.cc:1034] running on GCE node
I20240503 14:00:14.009856 18468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:14.010519 18468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:14.011687 18468 hybrid_clock.cc:648] HybridClock initialized: now 1714744814011657 us; error 45 us; skew 500 ppm
I20240503 14:00:14.013839 18468 webserver.cc:460] Webserver started at http://127.0.99.132:41109/ using document root <none> and password file <none>
I20240503 14:00:14.014489 18468 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:14.014549 18468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:14.014739 18468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:14.016641 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:14.017165 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:14.020977 18468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:00:14.023542 18483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:14.024935 18468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:14.025059 18468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:14.025179 18468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:14.058158 18468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:14.058874 18468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:14.059052 18468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:14.059665 18468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:14.060717 18468 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:14.060772 18468 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:14.060815 18468 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:14.060848 18468 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:14.069707 18468 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36471
I20240503 14:00:14.069731 18596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36471 every 8 connection(s)
I20240503 14:00:14.070752 18468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:00:14.078294 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 18468
I20240503 14:00:14.078436 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:00:14.080195 18597 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:14.080487 18597 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:14.081101 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:14.081108 18597 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:14.125087 18012 ts_manager.cc:194] Registered new tserver with Master: 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471)
I20240503 14:00:14.126099 18012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55743
W20240503 14:00:14.202105 18600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:14.202414 18600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:14.202486 18600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:14.207140 18600 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:14.207231 18600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:14.207319 18600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 14:00:14.211545 18600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.18600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:14.212608 18600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:14.213866 18600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:14.223918 18606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:14.224640 18607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:14.225350 18609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:14.226328 18600 server_base.cc:1034] running on GCE node
I20240503 14:00:14.226811 18600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:14.227478 18600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:14.228632 18600 hybrid_clock.cc:648] HybridClock initialized: now 1714744814228616 us; error 35 us; skew 500 ppm
I20240503 14:00:14.231142 18600 webserver.cc:460] Webserver started at http://127.0.99.133:40355/ using document root <none> and password file <none>
I20240503 14:00:14.231752 18600 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:14.231812 18600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:14.231997 18600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:14.234088 18600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "344bdf13092a447da55bd7a037da918a"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:14.234695 18600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "344bdf13092a447da55bd7a037da918a"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:14.238708 18600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:00:14.241225 18615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:14.242512 18600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:14.242666 18600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "344bdf13092a447da55bd7a037da918a"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:14.242772 18600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:14.264528 18600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:14.265290 18600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:14.265480 18600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:14.266139 18600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:14.267154 18600 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:14.267206 18600 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:14.267266 18600 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:14.267293 18600 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:14.275933 18600 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39929
I20240503 14:00:14.275956 18728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39929 every 8 connection(s)
I20240503 14:00:14.277052 18600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:00:14.277554 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 18600
I20240503 14:00:14.277701 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:00:14.286882 18729 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:14.287180 18729 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:14.287799 18729 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:14.323469 18012 ts_manager.cc:194] Registered new tserver with Master: 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929)
I20240503 14:00:14.324225 18012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:34217
I20240503 14:00:14.329587 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:00:14.340638 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:00:14.340770 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 18468
I20240503 14:00:14.345795 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 18600
I20240503 14:00:14.504292 18201 heartbeater.cc:503] Master 127.0.99.190:35671 was elected leader, sending a full tablet report...
I20240503 14:00:14.714145 18333 heartbeater.cc:503] Master 127.0.99.190:35671 was elected leader, sending a full tablet report...
I20240503 14:00:14.914243 18465 heartbeater.cc:503] Master 127.0.99.190:35671 was elected leader, sending a full tablet report...
I20240503 14:00:18.102849 18012 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37780:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:00:18.106433 18012 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:18.126168 18398 tablet_service.cc:1467] Processing CreateTablet for tablet 500961d6706043ca9bd3fd2eb3f274a4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.126175 18266 tablet_service.cc:1467] Processing CreateTablet for tablet 500961d6706043ca9bd3fd2eb3f274a4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.126961 18265 tablet_service.cc:1467] Processing CreateTablet for tablet bc3cae9382254940bc7bb1388d05bd82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.126809 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 30e32918970a4daa9d27134dfaa65a24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.127288 18266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500961d6706043ca9bd3fd2eb3f274a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.126168 18399 tablet_service.cc:1467] Processing CreateTablet for tablet 30e32918970a4daa9d27134dfaa65a24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.127635 18267 tablet_service.cc:1467] Processing CreateTablet for tablet 30e32918970a4daa9d27134dfaa65a24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.127837 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e32918970a4daa9d27134dfaa65a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.127857 18267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e32918970a4daa9d27134dfaa65a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.128000 18397 tablet_service.cc:1467] Processing CreateTablet for tablet bc3cae9382254940bc7bb1388d05bd82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.127288 18398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500961d6706043ca9bd3fd2eb3f274a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.126809 18133 tablet_service.cc:1467] Processing CreateTablet for tablet bc3cae9382254940bc7bb1388d05bd82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.128770 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cae9382254940bc7bb1388d05bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.128852 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cae9382254940bc7bb1388d05bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.129159 18134 tablet_service.cc:1467] Processing CreateTablet for tablet 500961d6706043ca9bd3fd2eb3f274a4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.129351 18399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e32918970a4daa9d27134dfaa65a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.129459 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500961d6706043ca9bd3fd2eb3f274a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.130230 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cae9382254940bc7bb1388d05bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.136873 18748 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:18.137010 18749 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:18.139374 18749 tablet_bootstrap.cc:654] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.139374 18748 tablet_bootstrap.cc:654] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.140300 18748 log.cc:824] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:18.140300 18749 log.cc:824] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:18.140480 18750 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:18.142343 18748 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:18.142343 18749 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:18.142557 18748 ts_tablet_manager.cc:1367] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:00:18.142558 18749 ts_tablet_manager.cc:1367] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:00:18.142699 18750 tablet_bootstrap.cc:654] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.143610 18750 log.cc:824] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:18.145337 18750 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:18.145542 18750 ts_tablet_manager.cc:1367] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:00:18.146579 18749 raft_consensus.cc:348] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.146579 18748 raft_consensus.cc:348] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.146847 18749 raft_consensus.cc:374] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.146847 18748 raft_consensus.cc:374] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.146907 18748 raft_consensus.cc:729] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.146907 18749 raft_consensus.cc:729] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.147429 18749 consensus_queue.cc:260] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.147429 18748 consensus_queue.cc:260] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.148614 18748 ts_tablet_manager.cc:1398] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 14:00:18.148614 18749 ts_tablet_manager.cc:1398] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 14:00:18.148927 18749 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:18.148932 18750 raft_consensus.cc:348] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.149189 18750 raft_consensus.cc:374] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.149250 18750 raft_consensus.cc:729] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.149757 18750 consensus_queue.cc:260] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.150068 18748 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:18.150090 18749 tablet_bootstrap.cc:654] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.150802 18750 ts_tablet_manager.cc:1398] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20240503 14:00:18.151108 18334 tablet.cc:2369] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:18.151427 18748 tablet_bootstrap.cc:654] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.151993 18750 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:18.153035 18750 tablet_bootstrap.cc:654] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.154372 18748 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:18.154486 18748 ts_tablet_manager.cc:1367] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:18.154587 18750 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:18.154721 18750 ts_tablet_manager.cc:1367] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:18.155170 18748 raft_consensus.cc:348] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.155318 18748 raft_consensus.cc:374] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.155361 18748 raft_consensus.cc:729] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.155479 18748 consensus_queue.cc:260] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.155509 18750 raft_consensus.cc:348] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.155718 18750 raft_consensus.cc:374] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.155791 18750 raft_consensus.cc:729] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.155874 18748 ts_tablet_manager.cc:1398] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:18.156049 18748 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:18.156006 18750 consensus_queue.cc:260] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.156052 18749 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:18.156144 18749 ts_tablet_manager.cc:1367] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:00:18.156523 18750 ts_tablet_manager.cc:1398] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:18.156751 18750 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:18.156783 18749 raft_consensus.cc:348] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.156893 18749 raft_consensus.cc:374] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.156925 18749 raft_consensus.cc:729] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.157104 18748 tablet_bootstrap.cc:654] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.157078 18749 consensus_queue.cc:260] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.157366 18749 ts_tablet_manager.cc:1398] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:18.158071 18750 tablet_bootstrap.cc:654] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.159133 18749 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:18.159248 18748 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:18.159344 18748 ts_tablet_manager.cc:1367] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:00:18.159806 18748 raft_consensus.cc:348] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.159925 18748 raft_consensus.cc:374] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.159960 18748 raft_consensus.cc:729] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.160074 18748 consensus_queue.cc:260] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.160341 18748 ts_tablet_manager.cc:1398] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.161515 18749 tablet_bootstrap.cc:654] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.162235 18750 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:18.162343 18750 ts_tablet_manager.cc:1367] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 14:00:18.162694 18749 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:18.162804 18749 ts_tablet_manager.cc:1367] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:00:18.162760 18750 raft_consensus.cc:348] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.162868 18750 raft_consensus.cc:374] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.162910 18750 raft_consensus.cc:729] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.163045 18750 consensus_queue.cc:260] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.163215 18749 raft_consensus.cc:348] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.163295 18750 ts_tablet_manager.cc:1398] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:18.163306 18749 raft_consensus.cc:374] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.163338 18749 raft_consensus.cc:729] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.163462 18749 consensus_queue.cc:260] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.163734 18749 ts_tablet_manager.cc:1398] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:18.195641 18756 raft_consensus.cc:482] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:18.195886 18756 raft_consensus.cc:504] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.197124 18756 leader_election.cc:288] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.201517 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e32918970a4daa9d27134dfaa65a24" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504" is_pre_election: true
I20240503 14:00:18.201905 18419 raft_consensus.cc:2454] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 0.
I20240503 14:00:18.202219 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e32918970a4daa9d27134dfaa65a24" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15" is_pre_election: true
I20240503 14:00:18.202500 18287 raft_consensus.cc:2454] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 0.
I20240503 14:00:18.202478 18089 leader_election.cc:302] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:18.202759 18756 raft_consensus.cc:2781] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:18.202860 18756 raft_consensus.cc:482] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:18.202896 18756 raft_consensus.cc:3037] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.204039 18756 raft_consensus.cc:504] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.204397 18756 leader_election.cc:288] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 election: Requested vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.204753 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e32918970a4daa9d27134dfaa65a24" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:18.204766 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e32918970a4daa9d27134dfaa65a24" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:18.204878 18419 raft_consensus.cc:3037] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.204901 18287 raft_consensus.cc:3037] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:00:18.205781 18202 tablet.cc:2369] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:18.205956 18287 raft_consensus.cc:2454] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 1.
I20240503 14:00:18.206297 18088 leader_election.cc:302] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ddc910d74d4e2eb9ef49440c1fb6ad, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.206331 18419 raft_consensus.cc:2454] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 1.
I20240503 14:00:18.206499 18756 raft_consensus.cc:2781] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:18.206601 18756 raft_consensus.cc:686] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Becoming Leader. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Running, Role: LEADER
I20240503 14:00:18.206933 18756 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.210628 18012 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:18.347671 18760 raft_consensus.cc:482] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:18.347857 18760 raft_consensus.cc:504] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.349057 18760 leader_election.cc:288] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.353832 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cae9382254940bc7bb1388d05bd82" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504" is_pre_election: true
I20240503 14:00:18.354053 18419 raft_consensus.cc:2454] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 0.
I20240503 14:00:18.354043 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cae9382254940bc7bb1388d05bd82" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:18.354310 18155 raft_consensus.cc:2454] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 0.
I20240503 14:00:18.354471 18221 leader_election.cc:302] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.354911 18760 raft_consensus.cc:2781] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:18.354988 18760 raft_consensus.cc:482] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:18.355026 18760 raft_consensus.cc:3037] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.355957 18760 raft_consensus.cc:504] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.356381 18760 leader_election.cc:288] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.356812 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cae9382254940bc7bb1388d05bd82" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:18.356824 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3cae9382254940bc7bb1388d05bd82" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:18.356961 18155 raft_consensus.cc:3037] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.356966 18419 raft_consensus.cc:3037] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.357800 18419 raft_consensus.cc:2454] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 1.
I20240503 14:00:18.357939 18155 raft_consensus.cc:2454] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 1.
I20240503 14:00:18.358121 18221 leader_election.cc:302] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.358350 18760 raft_consensus.cc:2781] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:18.358449 18760 raft_consensus.cc:686] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Becoming Leader. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Running, Role: LEADER
I20240503 14:00:18.358947 18760 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.362529 18012 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:18.364068 18466 tablet.cc:2369] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:18.380962 18760 raft_consensus.cc:482] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:18.381115 18760 raft_consensus.cc:504] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.381585 18760 leader_election.cc:288] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.382122 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500961d6706043ca9bd3fd2eb3f274a4" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:18.382192 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500961d6706043ca9bd3fd2eb3f274a4" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504" is_pre_election: true
I20240503 14:00:18.382294 18155 raft_consensus.cc:2454] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 0.
I20240503 14:00:18.382330 18419 raft_consensus.cc:2454] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 0.
I20240503 14:00:18.382607 18221 leader_election.cc:302] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.382819 18760 raft_consensus.cc:2781] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:18.382879 18760 raft_consensus.cc:482] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:18.382916 18760 raft_consensus.cc:3037] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.383806 18760 raft_consensus.cc:504] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.384209 18760 leader_election.cc:288] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.384729 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500961d6706043ca9bd3fd2eb3f274a4" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:18.384792 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500961d6706043ca9bd3fd2eb3f274a4" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:18.384861 18155 raft_consensus.cc:3037] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.384905 18419 raft_consensus.cc:3037] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.385663 18155 raft_consensus.cc:2454] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 1.
I20240503 14:00:18.385675 18419 raft_consensus.cc:2454] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 1.
I20240503 14:00:18.386034 18221 leader_election.cc:302] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.386231 18760 raft_consensus.cc:2781] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:18.386324 18760 raft_consensus.cc:686] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Becoming Leader. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Running, Role: LEADER
I20240503 14:00:18.386479 18760 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.388427 18012 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:18.538065 18772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:18.538363 18772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:18.541712 18772 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:00:18.640352 18287 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:18.641250 18419 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:18.641924 18761 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:18.642486 18756 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:18.645285 18418 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:18.651432 18155 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:18.652124 18754 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:18.653378 18418 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:18.654044 18760 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:18.655339 18754 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:18.658403 18154 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:18.662160 18754 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:18.663520 18802 mvcc.cc:205] Tried to move back new op lower bound from 7023594777140072448 to 7023594775378145280. Current Snapshot: MvccSnapshot[applied={T|T < 7023594777140072448}]
I20240503 14:00:18.666980 18804 mvcc.cc:205] Tried to move back new op lower bound from 7023594777140072448 to 7023594775378145280. Current Snapshot: MvccSnapshot[applied={T|T < 7023594777140072448}]
I20240503 14:00:18.692464 18797 mvcc.cc:205] Tried to move back new op lower bound from 7023594777190813696 to 7023594776001462272. Current Snapshot: MvccSnapshot[applied={T|T < 7023594777174405120}]
INSERT report
rows total: 2000
time total: 129.743 ms
time per row: 0.0648714 ms
I20240503 14:00:18.725358 18009 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37780:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:00:18.726024 18009 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:18.732162 18267 tablet_service.cc:1467] Processing CreateTablet for tablet ae4aca78536e4b97bdf12e0c41ea7614 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.732532 18267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4aca78536e4b97bdf12e0c41ea7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.732463 18134 tablet_service.cc:1467] Processing CreateTablet for tablet a163e1ed7c02469fb7c87e4953f0e517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.732638 18397 tablet_service.cc:1467] Processing CreateTablet for tablet ae4aca78536e4b97bdf12e0c41ea7614 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.732771 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163e1ed7c02469fb7c87e4953f0e517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.732887 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4aca78536e4b97bdf12e0c41ea7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.734656 18826 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:18.734882 18135 tablet_service.cc:1467] Processing CreateTablet for tablet ae4aca78536e4b97bdf12e0c41ea7614 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.735139 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4aca78536e4b97bdf12e0c41ea7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.735175 18398 tablet_service.cc:1467] Processing CreateTablet for tablet a163e1ed7c02469fb7c87e4953f0e517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.735417 18398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163e1ed7c02469fb7c87e4953f0e517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.736056 18826 tablet_bootstrap.cc:654] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.737349 18826 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:18.737445 18826 ts_tablet_manager.cc:1367] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:18.737455 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 149b0bffbe8a4a25afafd0f820f51b51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.737684 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b0bffbe8a4a25afafd0f820f51b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.737835 18266 tablet_service.cc:1467] Processing CreateTablet for tablet 149b0bffbe8a4a25afafd0f820f51b51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.737917 18826 raft_consensus.cc:348] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.738049 18826 raft_consensus.cc:374] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.738080 18826 raft_consensus.cc:729] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.738090 18266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b0bffbe8a4a25afafd0f820f51b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.738046 18398 tablet_service.cc:1467] Processing CreateTablet for tablet 149b0bffbe8a4a25afafd0f820f51b51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.738270 18398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b0bffbe8a4a25afafd0f820f51b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.738232 18826 consensus_queue.cc:260] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.740293 18827 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:18.740319 18826 ts_tablet_manager.cc:1398] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:18.740525 18826 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:18.740763 18267 tablet_service.cc:1467] Processing CreateTablet for tablet a163e1ed7c02469fb7c87e4953f0e517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:18.740973 18267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163e1ed7c02469fb7c87e4953f0e517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.741405 18826 tablet_bootstrap.cc:654] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.742149 18827 tablet_bootstrap.cc:654] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.742615 18826 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:18.742708 18826 ts_tablet_manager.cc:1367] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:18.743197 18827 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:18.743291 18826 raft_consensus.cc:348] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.743386 18827 ts_tablet_manager.cc:1367] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:18.743427 18826 raft_consensus.cc:374] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.743461 18826 raft_consensus.cc:729] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.743613 18826 consensus_queue.cc:260] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.743752 18828 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:18.743852 18827 raft_consensus.cc:348] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.743954 18827 raft_consensus.cc:374] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.743961 18826 ts_tablet_manager.cc:1398] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.743983 18827 raft_consensus.cc:729] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.744108 18826 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:18.744131 18827 consensus_queue.cc:260] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.744679 18828 tablet_bootstrap.cc:654] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.744817 18827 ts_tablet_manager.cc:1398] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.744975 18827 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:18.746090 18827 tablet_bootstrap.cc:654] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.746707 18826 tablet_bootstrap.cc:654] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.747797 18828 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:18.747820 18826 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:18.747884 18826 ts_tablet_manager.cc:1367] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:18.747895 18828 ts_tablet_manager.cc:1367] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:18.748310 18828 raft_consensus.cc:348] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.748318 18826 raft_consensus.cc:348] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.748421 18828 raft_consensus.cc:374] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.748430 18826 raft_consensus.cc:374] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.748454 18826 raft_consensus.cc:729] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.748452 18828 raft_consensus.cc:729] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.748558 18826 consensus_queue.cc:260] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.748582 18828 consensus_queue.cc:260] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.748795 18826 ts_tablet_manager.cc:1398] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.748850 18828 ts_tablet_manager.cc:1398] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.749027 18828 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:18.749583 18827 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:18.749675 18827 ts_tablet_manager.cc:1367] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:00:18.749938 18828 tablet_bootstrap.cc:654] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.750221 18827 raft_consensus.cc:348] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.750339 18827 raft_consensus.cc:374] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.750371 18827 raft_consensus.cc:729] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.750491 18827 consensus_queue.cc:260] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.750768 18827 ts_tablet_manager.cc:1398] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.750947 18827 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:18.751410 18828 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:18.751540 18828 ts_tablet_manager.cc:1367] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:18.751880 18827 tablet_bootstrap.cc:654] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.752441 18828 raft_consensus.cc:348] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.752552 18828 raft_consensus.cc:374] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.752590 18828 raft_consensus.cc:729] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.752879 18828 consensus_queue.cc:260] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.753232 18828 ts_tablet_manager.cc:1398] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:18.753418 18828 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:18.754339 18828 tablet_bootstrap.cc:654] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:18.754899 18827 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:18.754997 18827 ts_tablet_manager.cc:1367] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:00:18.755424 18828 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:18.755514 18828 ts_tablet_manager.cc:1367] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:18.755448 18827 raft_consensus.cc:348] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.755549 18827 raft_consensus.cc:374] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.755582 18827 raft_consensus.cc:729] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.755712 18827 consensus_queue.cc:260] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.755975 18827 ts_tablet_manager.cc:1398] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:18.755992 18828 raft_consensus.cc:348] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.756098 18828 raft_consensus.cc:374] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:18.756129 18828 raft_consensus.cc:729] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.756292 18828 consensus_queue.cc:260] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.756945 18828 ts_tablet_manager.cc:1398] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.794034 18825 raft_consensus.cc:482] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:18.794214 18825 raft_consensus.cc:504] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.794697 18825 leader_election.cc:288] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.795173 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b0bffbe8a4a25afafd0f820f51b51" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:18.795317 18154 raft_consensus.cc:2454] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 0.
I20240503 14:00:18.795302 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b0bffbe8a4a25afafd0f820f51b51" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504" is_pre_election: true
I20240503 14:00:18.795485 18418 raft_consensus.cc:2454] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 0.
I20240503 14:00:18.795606 18221 leader_election.cc:302] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ddc910d74d4e2eb9ef49440c1fb6ad, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.795799 18825 raft_consensus.cc:2781] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:18.795858 18825 raft_consensus.cc:482] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:18.795894 18825 raft_consensus.cc:3037] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.796748 18825 raft_consensus.cc:504] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.797084 18825 leader_election.cc:288] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.797549 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b0bffbe8a4a25afafd0f820f51b51" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:18.797639 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b0bffbe8a4a25afafd0f820f51b51" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:18.797690 18154 raft_consensus.cc:3037] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.797771 18418 raft_consensus.cc:3037] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.798631 18154 raft_consensus.cc:2454] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 1.
I20240503 14:00:18.798807 18418 raft_consensus.cc:2454] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 1.
I20240503 14:00:18.798943 18221 leader_election.cc:302] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ddc910d74d4e2eb9ef49440c1fb6ad, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.799124 18825 raft_consensus.cc:2781] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:18.799216 18825 raft_consensus.cc:686] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Becoming Leader. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Running, Role: LEADER
I20240503 14:00:18.799357 18825 consensus_queue.cc:237] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.801132 18012 catalog_manager.cc:5484] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:18.893918 18756 raft_consensus.cc:482] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:18.894099 18756 raft_consensus.cc:504] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.894582 18756 leader_election.cc:288] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.895097 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163e1ed7c02469fb7c87e4953f0e517" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504" is_pre_election: true
I20240503 14:00:18.895112 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163e1ed7c02469fb7c87e4953f0e517" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15" is_pre_election: true
I20240503 14:00:18.895265 18418 raft_consensus.cc:2454] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 0.
I20240503 14:00:18.895311 18287 raft_consensus.cc:2454] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 0.
I20240503 14:00:18.895648 18089 leader_election.cc:302] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:18.895874 18756 raft_consensus.cc:2781] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:18.895947 18756 raft_consensus.cc:482] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:18.895987 18756 raft_consensus.cc:3037] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.896903 18756 raft_consensus.cc:504] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.897315 18756 leader_election.cc:288] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 election: Requested vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:18.897635 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163e1ed7c02469fb7c87e4953f0e517" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:18.897765 18287 raft_consensus.cc:3037] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.897769 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a163e1ed7c02469fb7c87e4953f0e517" candidate_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:18.897879 18418 raft_consensus.cc:3037] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.898790 18287 raft_consensus.cc:2454] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 1.
I20240503 14:00:18.898792 18418 raft_consensus.cc:2454] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ddc910d74d4e2eb9ef49440c1fb6ad in term 1.
I20240503 14:00:18.899127 18088 leader_election.cc:302] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ddc910d74d4e2eb9ef49440c1fb6ad, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:18.899349 18756 raft_consensus.cc:2781] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:18.899443 18756 raft_consensus.cc:686] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Becoming Leader. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Running, Role: LEADER
I20240503 14:00:18.899592 18756 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.901490 18012 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:18.911641 18755 raft_consensus.cc:482] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:18.911798 18755 raft_consensus.cc:504] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.912899 18755 leader_election.cc:288] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:18.916391 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15" is_pre_election: true
I20240503 14:00:18.916455 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:18.916579 18154 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:18.916579 18287 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:18.916957 18353 leader_election.cc:302] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:18.917234 18755 raft_consensus.cc:2781] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:18.917301 18755 raft_consensus.cc:482] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:18.917346 18755 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.918355 18755 raft_consensus.cc:504] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.918722 18755 leader_election.cc:288] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Requested vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:18.919102 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:18.919229 18154 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.920040 18154 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:18.920284 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:18.920334 18353 leader_election.cc:302] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:18.920404 18287 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:18.920543 18755 raft_consensus.cc:2781] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:18.920785 18755 raft_consensus.cc:686] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Leader. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:18.921133 18755 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:18.921461 18287 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:18.924086 18012 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 625639f1ddfa446abaa3962865411504 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } }
W20240503 14:00:19.051165 18844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:19.051476 18844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:19.054699 18844 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:00:19.140854 18154 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.140827 18419 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.141587 18825 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.142053 18825 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.147049 18287 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.147462 18154 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.147605 18755 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.149264 18287 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.156188 18418 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.159332 18756 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.159852 18755 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.160547 18756 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.091 ms
time per row: 0.0550457 ms
I20240503 14:00:19.219386 18010 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37780:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:00:19.220010 18010 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:19.225752 18135 tablet_service.cc:1467] Processing CreateTablet for tablet dc81b3bd2aed42f69ae05df22854aaab (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.225795 18267 tablet_service.cc:1467] Processing CreateTablet for tablet dc81b3bd2aed42f69ae05df22854aaab (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.226142 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81b3bd2aed42f69ae05df22854aaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.226164 18267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81b3bd2aed42f69ae05df22854aaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.226300 18398 tablet_service.cc:1467] Processing CreateTablet for tablet dc81b3bd2aed42f69ae05df22854aaab (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.226552 18398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81b3bd2aed42f69ae05df22854aaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.226722 18266 tablet_service.cc:1467] Processing CreateTablet for tablet 1681d1aa0e244296a1457ea6faa48190 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.226938 18266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681d1aa0e244296a1457ea6faa48190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.228420 18827 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:19.228595 18828 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:19.228819 18397 tablet_service.cc:1467] Processing CreateTablet for tablet 1681d1aa0e244296a1457ea6faa48190 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.229030 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681d1aa0e244296a1457ea6faa48190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.228986 18265 tablet_service.cc:1467] Processing CreateTablet for tablet bb2ca36226704129a6522e683e6b9dda (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.229197 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca36226704129a6522e683e6b9dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.229313 18827 tablet_bootstrap.cc:654] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.229393 18828 tablet_bootstrap.cc:654] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.230022 18826 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:19.230757 18826 tablet_bootstrap.cc:654] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.231479 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 1681d1aa0e244296a1457ea6faa48190 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.231747 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681d1aa0e244296a1457ea6faa48190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.232121 18134 tablet_service.cc:1467] Processing CreateTablet for tablet bb2ca36226704129a6522e683e6b9dda (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.232218 18397 tablet_service.cc:1467] Processing CreateTablet for tablet bb2ca36226704129a6522e683e6b9dda (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:19.232414 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca36226704129a6522e683e6b9dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.232512 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca36226704129a6522e683e6b9dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:19.233705 18826 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:19.233817 18826 ts_tablet_manager.cc:1367] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:00:19.234367 18826 raft_consensus.cc:348] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.234489 18826 raft_consensus.cc:374] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.234520 18826 raft_consensus.cc:729] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.234643 18826 consensus_queue.cc:260] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.234931 18826 ts_tablet_manager.cc:1398] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:19.235105 18826 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:19.235293 18827 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:19.235397 18827 ts_tablet_manager.cc:1367] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:00:19.235831 18827 raft_consensus.cc:348] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.235941 18827 raft_consensus.cc:374] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.235980 18827 raft_consensus.cc:729] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.236137 18826 tablet_bootstrap.cc:654] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.236100 18827 consensus_queue.cc:260] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.236367 18827 ts_tablet_manager.cc:1398] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:19.236524 18827 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:19.237545 18827 tablet_bootstrap.cc:654] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.237705 18826 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:19.237809 18826 ts_tablet_manager.cc:1367] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:19.238675 18827 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:19.238760 18827 ts_tablet_manager.cc:1367] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:19.239233 18827 raft_consensus.cc:348] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.239354 18827 raft_consensus.cc:374] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.239390 18827 raft_consensus.cc:729] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.239512 18827 consensus_queue.cc:260] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.239620 18826 raft_consensus.cc:348] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.239763 18826 raft_consensus.cc:374] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.239800 18826 raft_consensus.cc:729] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.239897 18827 ts_tablet_manager.cc:1398] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:19.239953 18826 consensus_queue.cc:260] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.240065 18827 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:19.240286 18826 ts_tablet_manager.cc:1398] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:19.240444 18826 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:19.241111 18827 tablet_bootstrap.cc:654] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.242028 18826 tablet_bootstrap.cc:654] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.242195 18828 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:19.242336 18828 ts_tablet_manager.cc:1367] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:00:19.242852 18828 raft_consensus.cc:348] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.243005 18828 raft_consensus.cc:374] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.243041 18828 raft_consensus.cc:729] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.243191 18828 consensus_queue.cc:260] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.243382 18826 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: No bootstrap required, opened a new log
I20240503 14:00:19.243477 18826 ts_tablet_manager.cc:1367] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:00:19.243614 18828 ts_tablet_manager.cc:1398] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:19.243862 18826 raft_consensus.cc:348] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.243934 18828 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:19.243950 18826 raft_consensus.cc:374] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.243980 18826 raft_consensus.cc:729] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.244094 18826 consensus_queue.cc:260] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.244326 18826 ts_tablet_manager.cc:1398] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:19.244411 18827 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad: No bootstrap required, opened a new log
I20240503 14:00:19.244529 18827 ts_tablet_manager.cc:1367] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:19.244984 18828 tablet_bootstrap.cc:654] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.245057 18827 raft_consensus.cc:348] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.245182 18827 raft_consensus.cc:374] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.245229 18827 raft_consensus.cc:729] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.245379 18827 consensus_queue.cc:260] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.245725 18827 ts_tablet_manager.cc:1398] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:19.246380 18828 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:19.246518 18828 ts_tablet_manager.cc:1367] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:00:19.247054 18828 raft_consensus.cc:348] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.247220 18828 raft_consensus.cc:374] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.247272 18828 raft_consensus.cc:729] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.247419 18828 consensus_queue.cc:260] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.247928 18828 ts_tablet_manager.cc:1398] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:19.248119 18828 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Bootstrap starting.
I20240503 14:00:19.249214 18828 tablet_bootstrap.cc:654] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:19.250430 18828 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: No bootstrap required, opened a new log
I20240503 14:00:19.250514 18828 ts_tablet_manager.cc:1367] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:19.251169 18828 raft_consensus.cc:348] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.251272 18828 raft_consensus.cc:374] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:19.251294 18828 raft_consensus.cc:729] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Initialized, Role: FOLLOWER
I20240503 14:00:19.251412 18828 consensus_queue.cc:260] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.251677 18828 ts_tablet_manager.cc:1398] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:19.395208 18755 raft_consensus.cc:482] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:19.395363 18755 raft_consensus.cc:504] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.395794 18755 leader_election.cc:288] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:19.396356 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:19.396482 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15" is_pre_election: true
I20240503 14:00:19.396550 18154 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:19.396607 18286 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:19.396977 18353 leader_election.cc:302] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:19.397255 18755 raft_consensus.cc:2781] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:19.397322 18755 raft_consensus.cc:482] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:19.397367 18755 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.398380 18755 raft_consensus.cc:504] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.398766 18755 leader_election.cc:288] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:19.399202 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:19.399202 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:19.399341 18286 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.399341 18154 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.400337 18154 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:19.400337 18286 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:19.400730 18352 leader_election.cc:302] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:19.400921 18755 raft_consensus.cc:2781] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:19.400995 18755 raft_consensus.cc:686] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Leader. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:19.401134 18755 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.403059 18007 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 625639f1ddfa446abaa3962865411504 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:19.460095 18755 raft_consensus.cc:482] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:19.460243 18755 raft_consensus.cc:504] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.460675 18755 leader_election.cc:288] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:19.461201 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15" is_pre_election: true
I20240503 14:00:19.461201 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:19.461371 18154 raft_consensus.cc:2454] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:19.461371 18286 raft_consensus.cc:2454] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:19.461704 18352 leader_election.cc:302] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:19.461911 18755 raft_consensus.cc:2781] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:19.461997 18755 raft_consensus.cc:482] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:19.462047 18755 raft_consensus.cc:3037] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.462890 18755 raft_consensus.cc:504] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.463268 18755 leader_election.cc:288] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:19.463677 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:19.463696 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:19.463805 18154 raft_consensus.cc:3037] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.463837 18286 raft_consensus.cc:3037] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.464583 18154 raft_consensus.cc:2454] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:19.464819 18286 raft_consensus.cc:2454] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:19.464910 18353 leader_election.cc:302] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:19.465093 18755 raft_consensus.cc:2781] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:19.465238 18755 raft_consensus.cc:686] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Leader. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:19.465397 18755 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.467257 18007 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 625639f1ddfa446abaa3962865411504 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:19.525497 18755 raft_consensus.cc:482] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:19.525650 18755 raft_consensus.cc:504] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.526100 18755 leader_election.cc:288] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:19.526676 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15" is_pre_election: true
I20240503 14:00:19.526664 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" is_pre_election: true
I20240503 14:00:19.526835 18154 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:19.526835 18286 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 0.
I20240503 14:00:19.527169 18353 leader_election.cc:302] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:19.527400 18755 raft_consensus.cc:2781] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:19.527483 18755 raft_consensus.cc:482] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:19.527524 18755 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.530831 18755 raft_consensus.cc:504] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.531344 18755 leader_election.cc:288] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:19.531791 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:19.531793 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "625639f1ddfa446abaa3962865411504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:19.531929 18154 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.531929 18286 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:19.532766 18154 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:19.532773 18286 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 625639f1ddfa446abaa3962865411504 in term 1.
I20240503 14:00:19.533094 18353 leader_election.cc:302] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:19.533303 18755 raft_consensus.cc:2781] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:19.533403 18755 raft_consensus.cc:686] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Leader. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:19.533567 18755 consensus_queue.cc:237] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } }
I20240503 14:00:19.535419 18007 catalog_manager.cc:5484] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 625639f1ddfa446abaa3962865411504 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } }
W20240503 14:00:19.723735 18894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:19.724079 18894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:19.728577 18894 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:00:19.822106 18286 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.822106 18154 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.822495 18287 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.822510 18755 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.823102 18842 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.823504 18154 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.823843 18842 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.824029 18286 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.826285 18155 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:19.828227 18842 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.829209 18842 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:19.830538 18842 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.685 ms
time per row: 0.0518423 ms
I20240503 14:00:19.881889 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:36471
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:35671
--webserver_port=41109
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:00:20.013242 18948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:20.013478 18948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:20.013526 18948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:20.016865 18948 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:20.016953 18948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:20.017027 18948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 14:00:20.021102 18948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:36471
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=41109
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.18948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:20.022136 18948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:20.023332 18948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:20.033470 18953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:20.034152 18954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:20.035341 18956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:20.036195 18948 server_base.cc:1034] running on GCE node
I20240503 14:00:20.036746 18948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:20.037431 18948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:20.038615 18948 hybrid_clock.cc:648] HybridClock initialized: now 1714744820038589 us; error 41 us; skew 500 ppm
I20240503 14:00:20.041345 18948 webserver.cc:460] Webserver started at http://127.0.99.132:41109/ using document root <none> and password file <none>
I20240503 14:00:20.042063 18948 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:20.042156 18948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:20.046175 18948 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:00:20.048409 18962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:20.049665 18948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:20.049842 18948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:20.050376 18948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:20.083007 18948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:20.083863 18948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:20.084091 18948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:20.084786 18948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:20.085840 18948 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:20.085899 18948 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:20.085963 18948 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:20.086026 18948 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:20.095603 18948 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36471
I20240503 14:00:20.095625 19075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36471 every 8 connection(s)
I20240503 14:00:20.096848 18948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:00:20.103202 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 18948
I20240503 14:00:20.103659 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
/tmp/dist-test-taskmJWZN5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:39929
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:35671
--webserver_port=40355
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:00:20.106810 19076 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:20.107087 19076 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:20.107721 19076 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:20.109011 18013 ts_manager.cc:194] Re-registered known tserver with Master: 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471)
I20240503 14:00:20.109622 18013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:46985
W20240503 14:00:20.221693 19079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:20.222008 19079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:20.222074 19079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:20.225487 19079 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:00:20.225580 19079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:20.225675 19079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 14:00:20.229844 19079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:39929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40355
--tserver_master_addrs=127.0.99.190:35671
--never_fsync=true
--heap_profile_path=/tmp/kudu.19079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:20.230902 19079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:20.232272 19079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:20.242172 19084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:20.242760 19087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:20.243911 19085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:20.244717 19079 server_base.cc:1034] running on GCE node
I20240503 14:00:20.245204 19079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:20.245882 19079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:20.247061 19079 hybrid_clock.cc:648] HybridClock initialized: now 1714744820247040 us; error 38 us; skew 500 ppm
I20240503 14:00:20.249495 19079 webserver.cc:460] Webserver started at http://127.0.99.133:40355/ using document root <none> and password file <none>
I20240503 14:00:20.250206 19079 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:20.250281 19079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:20.253837 19079 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:20.256120 19093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:20.257256 19079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:20.257382 19079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "344bdf13092a447da55bd7a037da918a"
format_stamp: "Formatted at 2024-05-03 14:00:14 on dist-test-slave-6fwv"
I20240503 14:00:20.257871 19079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:20.283195 19079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:20.283954 19079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:20.284121 19079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:20.284734 19079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:20.285730 19079 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:20.285782 19079 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:20.285823 19079 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:20.285836 19079 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:20.294337 19079 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39929
I20240503 14:00:20.294370 19206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39929 every 8 connection(s)
I20240503 14:00:20.295487 19079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:00:20.300487 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu as pid 19079
I20240503 14:00:20.306057 19207 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35671
I20240503 14:00:20.306326 19207 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:20.306972 19207 heartbeater.cc:511] Master 127.0.99.190:35671 requested a full tablet report, sending...
I20240503 14:00:20.308344 18013 ts_manager.cc:194] Re-registered known tserver with Master: 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929)
I20240503 14:00:20.309171 18013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44607
I20240503 14:00:20.505102 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.505147 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.507087 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:20.508734 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.517755 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:20.611349 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:20.612932 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:20.613693 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.613737 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.623628 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.740393 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.742617 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.742885 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:20.743273 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.745702 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:20.842445 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:20.842761 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.843230 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.847561 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.855417 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:20.897450 18419 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:20.899423 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.899935 19355 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.900153 18155 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.900604 19355 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.902673 19354 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
W20240503 14:00:20.903898 18353 consensus_peers.cc:485] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 -> Peer 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471): Couldn't send request to peer 2911b16ff0c74be48ae3a6eaa3db27cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae4aca78536e4b97bdf12e0c41ea7614. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:20.903525 18155 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.903841 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.906253 18013 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:20.915449 18155 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:20.917202 18286 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.917639 18419 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.918208 19370 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.918478 19369 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.920825 18756 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.921211 18419 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
W20240503 14:00:20.922116 18089 consensus_peers.cc:485] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30e32918970a4daa9d27134dfaa65a24. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:20.923471 18286 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.924499 18005 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:20.933319 18419 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:20.935043 18155 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:20.935138 18353 consensus_peers.cc:485] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 -> Peer 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471): Couldn't send request to peer 2911b16ff0c74be48ae3a6eaa3db27cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb2ca36226704129a6522e683e6b9dda. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:20.935462 18286 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.935449 19354 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.935861 19354 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.937443 19355 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.937918 18155 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.938426 18286 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.940554 18013 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:20.949008 18419 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:20.950841 18286 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.951125 18155 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.951380 18947 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.951761 19355 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.953773 18947 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.954457 18155 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
W20240503 14:00:20.954888 18353 consensus_peers.cc:485] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc81b3bd2aed42f69ae05df22854aaab. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:20.954813 18285 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.956652 18010 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:20.967309 18285 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:20.969365 18419 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.969365 18155 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.969868 19397 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.970186 18760 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.972666 19396 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
W20240503 14:00:20.973102 18221 consensus_peers.cc:485] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 -> Peer 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471): Couldn't send request to peer 2911b16ff0c74be48ae3a6eaa3db27cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc3cae9382254940bc7bb1388d05bd82. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:20.973145 18419 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.973194 18155 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.976044 18007 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index -1 to 4, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:20.985231 18155 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:20.986934 18419 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:20.987064 18285 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:20.987254 18089 consensus_peers.cc:485] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a163e1ed7c02469fb7c87e4953f0e517. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:20.987495 18756 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.987720 19370 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:20.989737 19369 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.990408 18285 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.991688 18419 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:20.992578 18010 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index -1 to 4, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:21.111873 19076 heartbeater.cc:503] Master 127.0.99.190:35671 was elected leader, sending a full tablet report...
I20240503 14:00:21.311679 19207 heartbeater.cc:503] Master 127.0.99.190:35671 was elected leader, sending a full tablet report...
I20240503 14:00:21.351186 19469 ts_tablet_manager.cc:902] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:21.351991 19469 tablet_copy_client.cc:308] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:33181
I20240503 14:00:21.355669 19475 ts_tablet_manager.cc:902] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Initiating tablet copy from peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:21.356186 18439 tablet_copy_service.cc:140] P 625639f1ddfa446abaa3962865411504: Received BeginTabletCopySession request for tablet dc81b3bd2aed42f69ae05df22854aaab from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:57245)
I20240503 14:00:21.356377 18439 tablet_copy_service.cc:161] P 625639f1ddfa446abaa3962865411504: Beginning new tablet copy session on tablet dc81b3bd2aed42f69ae05df22854aaab from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:57245: session id = 344bdf13092a447da55bd7a037da918a-dc81b3bd2aed42f69ae05df22854aaab
I20240503 14:00:21.356503 19475 tablet_copy_client.cc:308] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36605
I20240503 14:00:21.357863 18439 tablet_copy_source_session.cc:205] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:21.360949 18307 tablet_copy_service.cc:140] P aaa289c674d34661bf49dd8bd7186a15: Received BeginTabletCopySession request for tablet bc3cae9382254940bc7bb1388d05bd82 from peer 2911b16ff0c74be48ae3a6eaa3db27cd ({username='slave'} at 127.0.99.132:59599)
I20240503 14:00:21.361117 18307 tablet_copy_service.cc:161] P aaa289c674d34661bf49dd8bd7186a15: Beginning new tablet copy session on tablet bc3cae9382254940bc7bb1388d05bd82 from peer 2911b16ff0c74be48ae3a6eaa3db27cd at {username='slave'} at 127.0.99.132:59599: session id = 2911b16ff0c74be48ae3a6eaa3db27cd-bc3cae9382254940bc7bb1388d05bd82
I20240503 14:00:21.362181 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81b3bd2aed42f69ae05df22854aaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:21.362540 18307 tablet_copy_source_session.cc:205] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:21.366475 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cae9382254940bc7bb1388d05bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:21.369876 19469 tablet_copy_client.cc:783] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:21.370141 19469 tablet_copy_client.cc:647] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:21.372231 19469 tablet_copy_client.cc:523] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:21.373281 19475 tablet_copy_client.cc:783] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:21.373490 19475 tablet_copy_client.cc:647] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:21.374325 19469 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:21.375449 19475 tablet_copy_client.cc:523] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:21.377568 19475 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap starting.
I20240503 14:00:21.381565 19469 log.cc:824] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:21.385363 19475 log.cc:824] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:21.411883 19469 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:21.412333 19469 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:21.412755 19469 ts_tablet_manager.cc:1367] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.039s user 0.032s sys 0.004s
I20240503 14:00:21.414253 19475 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:21.414638 19475 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap complete.
I20240503 14:00:21.414991 19475 ts_tablet_manager.cc:1367] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent bootstrapping tablet: real 0.038s user 0.033s sys 0.006s
I20240503 14:00:21.416265 19469 raft_consensus.cc:348] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:21.416620 19469 raft_consensus.cc:729] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:21.417049 19469 consensus_queue.cc:260] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:21.417687 19475 raft_consensus.cc:348] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:21.417937 19475 raft_consensus.cc:729] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2911b16ff0c74be48ae3a6eaa3db27cd, State: Initialized, Role: LEARNER
I20240503 14:00:21.418169 19469 ts_tablet_manager.cc:1398] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240503 14:00:21.418316 19475 consensus_queue.cc:260] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:21.419119 18439 tablet_copy_service.cc:342] P 625639f1ddfa446abaa3962865411504: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-dc81b3bd2aed42f69ae05df22854aaab received from {username='slave'} at 127.0.99.133:57245
I20240503 14:00:21.419227 18439 tablet_copy_service.cc:433] P 625639f1ddfa446abaa3962865411504: ending tablet copy session 344bdf13092a447da55bd7a037da918a-dc81b3bd2aed42f69ae05df22854aaab on tablet dc81b3bd2aed42f69ae05df22854aaab with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:21.419252 19475 ts_tablet_manager.cc:1398] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 14:00:21.419920 18307 tablet_copy_service.cc:342] P aaa289c674d34661bf49dd8bd7186a15: Request end of tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-bc3cae9382254940bc7bb1388d05bd82 received from {username='slave'} at 127.0.99.132:59599
I20240503 14:00:21.420008 18307 tablet_copy_service.cc:433] P aaa289c674d34661bf49dd8bd7186a15: ending tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-bc3cae9382254940bc7bb1388d05bd82 on tablet bc3cae9382254940bc7bb1388d05bd82 with peer 2911b16ff0c74be48ae3a6eaa3db27cd
I20240503 14:00:21.432792 19475 ts_tablet_manager.cc:902] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: Initiating tablet copy from peer 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:21.433275 19475 tablet_copy_client.cc:308] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:33181
I20240503 14:00:21.436683 18439 tablet_copy_service.cc:140] P 625639f1ddfa446abaa3962865411504: Received BeginTabletCopySession request for tablet ae4aca78536e4b97bdf12e0c41ea7614 from peer 2911b16ff0c74be48ae3a6eaa3db27cd ({username='slave'} at 127.0.99.132:35733)
I20240503 14:00:21.436815 18439 tablet_copy_service.cc:161] P 625639f1ddfa446abaa3962865411504: Beginning new tablet copy session on tablet ae4aca78536e4b97bdf12e0c41ea7614 from peer 2911b16ff0c74be48ae3a6eaa3db27cd at {username='slave'} at 127.0.99.132:35733: session id = 2911b16ff0c74be48ae3a6eaa3db27cd-ae4aca78536e4b97bdf12e0c41ea7614
I20240503 14:00:21.437587 18439 tablet_copy_source_session.cc:205] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:21.438609 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4aca78536e4b97bdf12e0c41ea7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:21.439992 19489 ts_tablet_manager.cc:902] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: Initiating tablet copy from peer 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:21.440547 19489 tablet_copy_client.cc:308] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:33181
I20240503 14:00:21.440878 19475 tablet_copy_client.cc:783] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:21.441046 19475 tablet_copy_client.cc:647] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:21.441098 18439 tablet_copy_service.cc:140] P 625639f1ddfa446abaa3962865411504: Received BeginTabletCopySession request for tablet bb2ca36226704129a6522e683e6b9dda from peer 2911b16ff0c74be48ae3a6eaa3db27cd ({username='slave'} at 127.0.99.132:35733)
I20240503 14:00:21.441197 18439 tablet_copy_service.cc:161] P 625639f1ddfa446abaa3962865411504: Beginning new tablet copy session on tablet bb2ca36226704129a6522e683e6b9dda from peer 2911b16ff0c74be48ae3a6eaa3db27cd at {username='slave'} at 127.0.99.132:35733: session id = 2911b16ff0c74be48ae3a6eaa3db27cd-bb2ca36226704129a6522e683e6b9dda
I20240503 14:00:21.441967 18439 tablet_copy_source_session.cc:205] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:21.442346 19475 tablet_copy_client.cc:523] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:21.442746 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca36226704129a6522e683e6b9dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:21.443444 19475 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap starting.
I20240503 14:00:21.444528 19489 tablet_copy_client.cc:783] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:21.444679 19489 tablet_copy_client.cc:647] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:21.445822 19489 tablet_copy_client.cc:523] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:21.447125 19489 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap starting.
I20240503 14:00:21.466463 19469 ts_tablet_manager.cc:902] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:21.466965 19469 tablet_copy_client.cc:308] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33467
I20240503 14:00:21.470937 18175 tablet_copy_service.cc:140] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Received BeginTabletCopySession request for tablet 30e32918970a4daa9d27134dfaa65a24 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:53929)
I20240503 14:00:21.471108 18175 tablet_copy_service.cc:161] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Beginning new tablet copy session on tablet 30e32918970a4daa9d27134dfaa65a24 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:53929: session id = 344bdf13092a447da55bd7a037da918a-30e32918970a4daa9d27134dfaa65a24
I20240503 14:00:21.471220 19475 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:21.471800 19475 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap complete.
I20240503 14:00:21.472199 19475 ts_tablet_manager.cc:1367] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.004s
I20240503 14:00:21.472605 18175 tablet_copy_source_session.cc:205] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:21.472831 19475 raft_consensus.cc:348] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:21.472954 19475 raft_consensus.cc:729] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2911b16ff0c74be48ae3a6eaa3db27cd, State: Initialized, Role: LEARNER
I20240503 14:00:21.473050 19475 consensus_queue.cc:260] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:21.473311 19475 ts_tablet_manager.cc:1398] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:21.474462 18439 tablet_copy_service.cc:342] P 625639f1ddfa446abaa3962865411504: Request end of tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-ae4aca78536e4b97bdf12e0c41ea7614 received from {username='slave'} at 127.0.99.132:35733
I20240503 14:00:21.474548 18439 tablet_copy_service.cc:433] P 625639f1ddfa446abaa3962865411504: ending tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-ae4aca78536e4b97bdf12e0c41ea7614 on tablet ae4aca78536e4b97bdf12e0c41ea7614 with peer 2911b16ff0c74be48ae3a6eaa3db27cd
I20240503 14:00:21.474776 19489 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:21.475088 19489 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap complete.
I20240503 14:00:21.475265 19489 ts_tablet_manager.cc:1367] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.004s
I20240503 14:00:21.475622 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e32918970a4daa9d27134dfaa65a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:21.476162 19489 raft_consensus.cc:348] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:21.476290 19489 raft_consensus.cc:729] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2911b16ff0c74be48ae3a6eaa3db27cd, State: Initialized, Role: LEARNER
I20240503 14:00:21.476395 19489 consensus_queue.cc:260] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:21.476676 19489 ts_tablet_manager.cc:1398] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:21.477591 19469 tablet_copy_client.cc:783] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:21.477725 19469 tablet_copy_client.cc:647] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:21.478358 18439 tablet_copy_service.cc:342] P 625639f1ddfa446abaa3962865411504: Request end of tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-bb2ca36226704129a6522e683e6b9dda received from {username='slave'} at 127.0.99.132:35733
I20240503 14:00:21.478452 18439 tablet_copy_service.cc:433] P 625639f1ddfa446abaa3962865411504: ending tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-bb2ca36226704129a6522e683e6b9dda on tablet bb2ca36226704129a6522e683e6b9dda with peer 2911b16ff0c74be48ae3a6eaa3db27cd
I20240503 14:00:21.480073 19469 tablet_copy_client.cc:523] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:21.481439 19469 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:21.502434 19469 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:21.502679 19469 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:21.502843 19469 ts_tablet_manager.cc:1367] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.004s
I20240503 14:00:21.503369 19469 raft_consensus.cc:348] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:21.503486 19469 raft_consensus.cc:729] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:21.503569 19469 consensus_queue.cc:260] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:21.503799 19469 ts_tablet_manager.cc:1398] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:21.504457 18175 tablet_copy_service.cc:342] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-30e32918970a4daa9d27134dfaa65a24 received from {username='slave'} at 127.0.99.133:53929
I20240503 14:00:21.504572 18175 tablet_copy_service.cc:433] P a9ddc910d74d4e2eb9ef49440c1fb6ad: ending tablet copy session 344bdf13092a447da55bd7a037da918a-30e32918970a4daa9d27134dfaa65a24 on tablet 30e32918970a4daa9d27134dfaa65a24 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:21.550558 19469 ts_tablet_manager.cc:902] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:21.551074 19469 tablet_copy_client.cc:308] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33467
I20240503 14:00:21.551553 18175 tablet_copy_service.cc:140] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Received BeginTabletCopySession request for tablet a163e1ed7c02469fb7c87e4953f0e517 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:53929)
I20240503 14:00:21.551667 18175 tablet_copy_service.cc:161] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Beginning new tablet copy session on tablet a163e1ed7c02469fb7c87e4953f0e517 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:53929: session id = 344bdf13092a447da55bd7a037da918a-a163e1ed7c02469fb7c87e4953f0e517
I20240503 14:00:21.552409 18175 tablet_copy_source_session.cc:205] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:21.553309 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163e1ed7c02469fb7c87e4953f0e517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:21.555320 19469 tablet_copy_client.cc:783] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:21.555482 19469 tablet_copy_client.cc:647] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:21.556682 19469 tablet_copy_client.cc:523] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:21.557811 19469 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:21.574874 19469 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:21.575168 19469 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:21.575361 19469 ts_tablet_manager.cc:1367] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240503 14:00:21.575845 19469 raft_consensus.cc:348] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:21.575969 19469 raft_consensus.cc:729] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:21.576053 19469 consensus_queue.cc:260] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:21.576315 19469 ts_tablet_manager.cc:1398] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:21.576948 18175 tablet_copy_service.cc:342] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-a163e1ed7c02469fb7c87e4953f0e517 received from {username='slave'} at 127.0.99.133:53929
I20240503 14:00:21.577045 18175 tablet_copy_service.cc:433] P a9ddc910d74d4e2eb9ef49440c1fb6ad: ending tablet copy session 344bdf13092a447da55bd7a037da918a-a163e1ed7c02469fb7c87e4953f0e517 on tablet a163e1ed7c02469fb7c87e4953f0e517 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:21.860549 19030 raft_consensus.cc:1203] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:21.908296 19161 raft_consensus.cc:1203] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:21.914278 19030 raft_consensus.cc:1203] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:21.923050 19161 raft_consensus.cc:1203] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:21.958882 19030 raft_consensus.cc:1203] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:22.140939 19161 raft_consensus.cc:1203] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:22.288599 19569 raft_consensus.cc:1050] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:22.289139 19569 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:22.290740 18418 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.290771 18286 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.290848 19161 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.291270 19632 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.291501 19569 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.291762 19369 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.294052 19369 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.296130 18286 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.296694 18013 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.298501 19161 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.298849 18418 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.304556 18155 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:22.306073 19161 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.306109 18418 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.307063 19632 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.307329 19569 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.309021 19369 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.309603 18418 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.309620 19161 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.310987 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30e32918970a4daa9d27134dfaa65a24 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:22.312296 18010 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.315228 18265 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e32918970a4daa9d27134dfaa65a24 with delete_type TABLET_DATA_TOMBSTONED (TS aaa289c674d34661bf49dd8bd7186a15 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58598
I20240503 14:00:22.315812 19640 tablet_replica.cc:330] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: stopping tablet replica
I20240503 14:00:22.316118 19640 raft_consensus.cc:2229] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.316429 19640 raft_consensus.cc:2258] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.317618 19640 ts_tablet_manager.cc:1875] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.319919 19640 ts_tablet_manager.cc:1888] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.320008 19640 log.cc:1196] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/30e32918970a4daa9d27134dfaa65a24
I20240503 14:00:22.320694 17997 catalog_manager.cc:4830] TS aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605): tablet 30e32918970a4daa9d27134dfaa65a24 (table rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]) successfully deleted
I20240503 14:00:22.388998 19641 raft_consensus.cc:1050] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: attempting to promote NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd to VOTER
I20240503 14:00:22.389465 19641 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:22.390982 18418 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.391033 18155 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.391084 19030 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.391479 19643 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.391697 19644 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.391866 19567 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.394192 19643 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.394678 18155 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.395637 19030 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.396958 18007 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.398835 18418 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.400671 18286 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:22.402416 19030 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.402567 18418 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.402819 19567 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.403052 19644 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.404772 19644 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.405308 18418 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.405398 19030 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.406863 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc3cae9382254940bc7bb1388d05bd82 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:22.407362 18007 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.410252 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3cae9382254940bc7bb1388d05bd82 with delete_type TABLET_DATA_TOMBSTONED (TS a9ddc910d74d4e2eb9ef49440c1fb6ad not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47852
I20240503 14:00:22.411263 19648 tablet_replica.cc:330] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: stopping tablet replica
I20240503 14:00:22.411628 19648 raft_consensus.cc:2229] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.411882 19648 raft_consensus.cc:2258] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.412873 19354 raft_consensus.cc:1050] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:22.413121 19648 ts_tablet_manager.cc:1875] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.413343 19354 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:22.414883 18155 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.415014 19161 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.415163 18286 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.415364 19570 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.415588 19650 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.416028 19355 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.416288 19648 ts_tablet_manager.cc:1888] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.416376 19648 log.cc:1196] T bc3cae9382254940bc7bb1388d05bd82 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bc3cae9382254940bc7bb1388d05bd82
I20240503 14:00:22.417008 17998 catalog_manager.cc:4830] TS a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467): tablet bc3cae9382254940bc7bb1388d05bd82 (table rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]) successfully deleted
I20240503 14:00:22.418542 19650 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.419027 18155 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.419039 19161 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.421022 18286 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.421541 18007 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.425025 18418 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:22.426398 19161 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.426427 18155 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.426862 19654 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.427084 19650 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.428869 19570 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.429368 19161 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.429522 18155 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.430774 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc81b3bd2aed42f69ae05df22854aaab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:22.431779 18005 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.435058 18265 tablet_service.cc:1514] Processing DeleteTablet for tablet dc81b3bd2aed42f69ae05df22854aaab with delete_type TABLET_DATA_TOMBSTONED (TS aaa289c674d34661bf49dd8bd7186a15 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58598
I20240503 14:00:22.435325 19640 tablet_replica.cc:330] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: stopping tablet replica
I20240503 14:00:22.435529 19640 raft_consensus.cc:2229] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.435715 19640 raft_consensus.cc:2258] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.436589 19640 ts_tablet_manager.cc:1875] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.439054 19640 ts_tablet_manager.cc:1888] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.439141 19640 log.cc:1196] T dc81b3bd2aed42f69ae05df22854aaab P aaa289c674d34661bf49dd8bd7186a15: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc81b3bd2aed42f69ae05df22854aaab
I20240503 14:00:22.439314 19654 raft_consensus.cc:1050] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: attempting to promote NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd to VOTER
I20240503 14:00:22.439349 19650 raft_consensus.cc:1050] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: attempting to promote NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd to VOTER
I20240503 14:00:22.439635 17997 catalog_manager.cc:4830] TS aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605): tablet dc81b3bd2aed42f69ae05df22854aaab (table rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]) successfully deleted
I20240503 14:00:22.439736 19650 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:22.439741 19654 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:22.441208 18286 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.441246 18155 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.441246 19029 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.441562 19355 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.442289 18155 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.442286 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.442641 19570 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.442916 19654 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.442970 19570 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.443320 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.443571 19354 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.444790 19570 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.446152 19654 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.446687 18155 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.448141 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.449142 19355 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.449395 18013 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.450656 18154 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.451795 18286 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.454233 19030 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.454411 19029 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.457374 18005 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd reported cstate change: config changed from index 4 to 5, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { replace: true } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.458688 18418 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:22.460004 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.460019 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.461053 19654 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.461248 19650 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.463069 18418 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:22.463301 19654 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.463667 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.464237 19029 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.464905 19030 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.465277 19650 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.465713 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae4aca78536e4b97bdf12e0c41ea7614 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:22.465726 18010 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.467566 19650 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.468082 18286 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.468498 19570 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.469372 19029 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.469511 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4aca78536e4b97bdf12e0c41ea7614 with delete_type TABLET_DATA_TOMBSTONED (TS a9ddc910d74d4e2eb9ef49440c1fb6ad not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47852
I20240503 14:00:22.469798 19648 tablet_replica.cc:330] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: stopping tablet replica
I20240503 14:00:22.469949 19648 raft_consensus.cc:2229] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.470134 19648 raft_consensus.cc:2258] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.470651 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb2ca36226704129a6522e683e6b9dda with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:22.471065 18286 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:22.471851 18007 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 5 to 6, VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.473193 19648 ts_tablet_manager.cc:1875] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.474999 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2ca36226704129a6522e683e6b9dda with delete_type TABLET_DATA_TOMBSTONED (TS a9ddc910d74d4e2eb9ef49440c1fb6ad not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47852
I20240503 14:00:22.475872 19648 ts_tablet_manager.cc:1888] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.475957 19648 log.cc:1196] T ae4aca78536e4b97bdf12e0c41ea7614 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae4aca78536e4b97bdf12e0c41ea7614
I20240503 14:00:22.476526 19648 tablet_replica.cc:330] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: stopping tablet replica
I20240503 14:00:22.476666 19648 raft_consensus.cc:2229] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.476742 17998 catalog_manager.cc:4830] TS a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467): tablet ae4aca78536e4b97bdf12e0c41ea7614 (table rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]) successfully deleted
I20240503 14:00:22.476827 19648 raft_consensus.cc:2258] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.477553 19648 ts_tablet_manager.cc:1875] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.479398 19648 ts_tablet_manager.cc:1888] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.479465 19648 log.cc:1196] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb2ca36226704129a6522e683e6b9dda
I20240503 14:00:22.479902 17998 catalog_manager.cc:4830] TS a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467): tablet bb2ca36226704129a6522e683e6b9dda (table rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]) successfully deleted
I20240503 14:00:22.559724 19632 raft_consensus.cc:1050] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:22.560213 19632 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:22.561677 19161 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.561707 18418 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.561707 18286 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.562125 19632 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.562316 19568 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.562486 19369 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.564452 19632 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.565048 19161 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.565127 18418 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.567181 18286 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.567528 18010 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 4 to 5, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.570407 18154 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:22.571578 19161 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.571616 18418 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.571933 19568 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.572126 19569 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.573565 19369 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.574010 18418 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.574074 19161 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:22.575115 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a163e1ed7c02469fb7c87e4953f0e517 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:22.576169 18005 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 5 to 6, VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:22.578689 18265 tablet_service.cc:1514] Processing DeleteTablet for tablet a163e1ed7c02469fb7c87e4953f0e517 with delete_type TABLET_DATA_TOMBSTONED (TS aaa289c674d34661bf49dd8bd7186a15 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58598
I20240503 14:00:22.578933 19640 tablet_replica.cc:330] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: stopping tablet replica
I20240503 14:00:22.579061 19640 raft_consensus.cc:2229] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.579197 19640 raft_consensus.cc:2258] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.579802 19640 ts_tablet_manager.cc:1875] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.581722 19640 ts_tablet_manager.cc:1888] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.581809 19640 log.cc:1196] T a163e1ed7c02469fb7c87e4953f0e517 P aaa289c674d34661bf49dd8bd7186a15: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a163e1ed7c02469fb7c87e4953f0e517
I20240503 14:00:22.582330 17997 catalog_manager.cc:4830] TS aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605): tablet a163e1ed7c02469fb7c87e4953f0e517 (table rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]) successfully deleted
I20240503 14:00:22.767241 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.767586 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.767863 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:22.767985 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:22.770538 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.866318 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.872112 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:22.878679 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.880352 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:22.890321 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.974892 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.976614 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.977142 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:22.977171 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:22.986181 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.179697 18005 ts_manager.cc:292] Set tserver state for 625639f1ddfa446abaa3962865411504 to MAINTENANCE_MODE
I20240503 14:00:23.399873 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.399974 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.401096 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:23.401955 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.412220 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:23.509174 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.510412 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:23.511078 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.511617 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.512053 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:23.650645 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.652238 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.654438 19141 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.655529 18397 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:23.658485 18265 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:23.701416 18286 consensus_queue.cc:237] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } }
I20240503 14:00:23.702881 18418 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:23.702890 18154 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:23.703347 19929 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:23.703563 19805 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:23.705435 19932 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.705886 18418 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.705950 18154 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.708812 18007 catalog_manager.cc:5484] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 14:00:23.712677 18286 consensus_queue.cc:237] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:23.715004 18418 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.715380 18154 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.715557 19932 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.715785 19929 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.717756 19929 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.718832 18286 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } }
I20240503 14:00:23.719122 18154 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.720144 17997 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 149b0bffbe8a4a25afafd0f820f51b51 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:23.720536 18418 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.722249 18419 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:23.723388 18007 catalog_manager.cc:5484] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.723961 19929 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:23.724386 18154 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:23.725566 18221 consensus_peers.cc:485] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 149b0bffbe8a4a25afafd0f820f51b51. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:23.725746 19943 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:23.728763 19932 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.730737 18419 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.731220 18154 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.731727 18010 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 14:00:23.735852 18286 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:23.737418 18154 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.737483 18419 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.737883 19943 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:23.738168 18221 consensus_peers.cc:485] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500961d6706043ca9bd3fd2eb3f274a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:23.738261 19929 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.740773 19932 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.742517 17997 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 500961d6706043ca9bd3fd2eb3f274a4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:23.742641 18154 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.742774 18418 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.744865 18007 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:23.745437 18419 consensus_queue.cc:237] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } }
I20240503 14:00:23.747210 18154 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:23.747208 18286 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:23.747633 19952 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:23.748284 19808 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:23.749483 19808 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.749919 18154 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.750751 18286 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.753175 18013 catalog_manager.cc:5484] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } }
I20240503 14:00:23.756557 18419 consensus_queue.cc:237] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:23.758046 18286 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.758119 18154 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:00:23.758332 18353 consensus_peers.cc:485] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1681d1aa0e244296a1457ea6faa48190. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:23.758517 19952 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.758728 19808 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.760443 19960 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.760922 18286 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.761905 18154 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:23.763518 17998 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1681d1aa0e244296a1457ea6faa48190 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:23.764566 18010 catalog_manager.cc:5484] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 4 to 5, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:24.134145 19987 ts_tablet_manager.cc:902] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:24.134631 19987 tablet_copy_client.cc:308] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36605
I20240503 14:00:24.138329 18307 tablet_copy_service.cc:140] P aaa289c674d34661bf49dd8bd7186a15: Received BeginTabletCopySession request for tablet 149b0bffbe8a4a25afafd0f820f51b51 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:58409)
I20240503 14:00:24.138470 18307 tablet_copy_service.cc:161] P aaa289c674d34661bf49dd8bd7186a15: Beginning new tablet copy session on tablet 149b0bffbe8a4a25afafd0f820f51b51 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:58409: session id = 344bdf13092a447da55bd7a037da918a-149b0bffbe8a4a25afafd0f820f51b51
I20240503 14:00:24.139355 18307 tablet_copy_source_session.cc:205] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:24.140304 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b0bffbe8a4a25afafd0f820f51b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:24.142406 19987 tablet_copy_client.cc:783] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:24.142598 19987 tablet_copy_client.cc:647] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:24.143889 19987 tablet_copy_client.cc:523] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:24.145071 19987 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:24.164568 19987 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:24.164888 19987 tablet_bootstrap.cc:492] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:24.165110 19987 ts_tablet_manager.cc:1367] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 14:00:24.165657 19987 raft_consensus.cc:348] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:24.165807 19987 raft_consensus.cc:729] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:24.165908 19987 consensus_queue.cc:260] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:24.166363 19987 ts_tablet_manager.cc:1398] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:00:24.166966 18307 tablet_copy_service.cc:342] P aaa289c674d34661bf49dd8bd7186a15: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-149b0bffbe8a4a25afafd0f820f51b51 received from {username='slave'} at 127.0.99.133:58409
I20240503 14:00:24.167052 18307 tablet_copy_service.cc:433] P aaa289c674d34661bf49dd8bd7186a15: ending tablet copy session 344bdf13092a447da55bd7a037da918a-149b0bffbe8a4a25afafd0f820f51b51 on tablet 149b0bffbe8a4a25afafd0f820f51b51 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:24.174770 19987 ts_tablet_manager.cc:902] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:24.175304 19987 tablet_copy_client.cc:308] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36605
I20240503 14:00:24.175803 18307 tablet_copy_service.cc:140] P aaa289c674d34661bf49dd8bd7186a15: Received BeginTabletCopySession request for tablet 500961d6706043ca9bd3fd2eb3f274a4 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:58409)
I20240503 14:00:24.175913 18307 tablet_copy_service.cc:161] P aaa289c674d34661bf49dd8bd7186a15: Beginning new tablet copy session on tablet 500961d6706043ca9bd3fd2eb3f274a4 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:58409: session id = 344bdf13092a447da55bd7a037da918a-500961d6706043ca9bd3fd2eb3f274a4
I20240503 14:00:24.176692 18307 tablet_copy_source_session.cc:205] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:24.177594 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500961d6706043ca9bd3fd2eb3f274a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:24.179483 19987 tablet_copy_client.cc:783] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:24.179642 19987 tablet_copy_client.cc:647] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:24.180830 19987 tablet_copy_client.cc:523] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:24.182011 19987 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:24.202081 19987 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:24.202401 19987 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:24.202656 19987 ts_tablet_manager.cc:1367] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.021s user 0.021s sys 0.000s
I20240503 14:00:24.203152 19987 raft_consensus.cc:348] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:24.203469 19987 raft_consensus.cc:729] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:24.203612 19987 consensus_queue.cc:260] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:24.203996 19987 ts_tablet_manager.cc:1398] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:24.204789 18307 tablet_copy_service.cc:342] P aaa289c674d34661bf49dd8bd7186a15: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-500961d6706043ca9bd3fd2eb3f274a4 received from {username='slave'} at 127.0.99.133:58409
I20240503 14:00:24.204895 18307 tablet_copy_service.cc:433] P aaa289c674d34661bf49dd8bd7186a15: ending tablet copy session 344bdf13092a447da55bd7a037da918a-500961d6706043ca9bd3fd2eb3f274a4 on tablet 500961d6706043ca9bd3fd2eb3f274a4 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:24.361192 19987 ts_tablet_manager.cc:902] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:24.361721 19987 tablet_copy_client.cc:308] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:33181
I20240503 14:00:24.362349 18439 tablet_copy_service.cc:140] P 625639f1ddfa446abaa3962865411504: Received BeginTabletCopySession request for tablet 1681d1aa0e244296a1457ea6faa48190 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:57245)
I20240503 14:00:24.362476 18439 tablet_copy_service.cc:161] P 625639f1ddfa446abaa3962865411504: Beginning new tablet copy session on tablet 1681d1aa0e244296a1457ea6faa48190 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:57245: session id = 344bdf13092a447da55bd7a037da918a-1681d1aa0e244296a1457ea6faa48190
I20240503 14:00:24.363322 18439 tablet_copy_source_session.cc:205] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:24.364284 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681d1aa0e244296a1457ea6faa48190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:24.366176 19987 tablet_copy_client.cc:783] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:24.366359 19987 tablet_copy_client.cc:647] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:24.367524 19987 tablet_copy_client.cc:523] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:24.368702 19987 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:24.385695 19987 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:24.386014 19987 tablet_bootstrap.cc:492] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:24.386198 19987 ts_tablet_manager.cc:1367] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240503 14:00:24.386683 19987 raft_consensus.cc:348] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:24.386804 19987 raft_consensus.cc:729] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:24.386904 19987 consensus_queue.cc:260] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:24.387154 19987 ts_tablet_manager.cc:1398] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:24.387827 18439 tablet_copy_service.cc:342] P 625639f1ddfa446abaa3962865411504: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-1681d1aa0e244296a1457ea6faa48190 received from {username='slave'} at 127.0.99.133:57245
I20240503 14:00:24.387912 18439 tablet_copy_service.cc:433] P 625639f1ddfa446abaa3962865411504: ending tablet copy session 344bdf13092a447da55bd7a037da918a-1681d1aa0e244296a1457ea6faa48190 on tablet 1681d1aa0e244296a1457ea6faa48190 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:24.641057 19161 raft_consensus.cc:1203] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:24.659873 19161 raft_consensus.cc:1203] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:24.958056 19161 raft_consensus.cc:1203] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:25.216789 20083 raft_consensus.cc:1050] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:25.217226 20083 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.218740 19161 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.218808 18154 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.218844 18419 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.219425 20086 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.219645 20085 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.219823 20083 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.222169 20085 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.222826 18154 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.223026 18419 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.225245 19161 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.225292 18013 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.229264 18286 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.230443 19161 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.230690 18154 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.230906 20086 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.231292 20086 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.232573 20086 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.233180 19161 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.233692 18154 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.234337 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 500961d6706043ca9bd3fd2eb3f274a4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:25.236066 18005 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.238569 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet 500961d6706043ca9bd3fd2eb3f274a4 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:25.239168 20094 tablet_replica.cc:330] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:25.239516 20094 raft_consensus.cc:2229] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:25.239734 20094 raft_consensus.cc:2258] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:25.240943 20094 ts_tablet_manager.cc:1875] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:25.243355 20094 ts_tablet_manager.cc:1888] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:00:25.243448 20094 log.cc:1196] T 500961d6706043ca9bd3fd2eb3f274a4 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/500961d6706043ca9bd3fd2eb3f274a4
I20240503 14:00:25.244066 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet 500961d6706043ca9bd3fd2eb3f274a4 (table rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]) successfully deleted
I20240503 14:00:25.283512 20086 raft_consensus.cc:1050] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:25.284035 20086 consensus_queue.cc:237] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.285605 18154 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.285634 18419 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.286067 19161 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.286072 20085 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.286331 19932 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.286481 20083 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.288750 20083 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.289254 18419 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.289312 19161 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.291518 18010 catalog_manager.cc:5484] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:25.291755 18154 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.294715 18286 consensus_queue.cc:237] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.296054 18154 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.296054 19161 raft_consensus.cc:1261] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.296553 20085 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.296818 20086 consensus_queue.cc:1035] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.298576 20085 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.298970 18154 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.299017 19161 raft_consensus.cc:2932] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.300843 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 149b0bffbe8a4a25afafd0f820f51b51 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:25.301337 18013 catalog_manager.cc:5484] T 149b0bffbe8a4a25afafd0f820f51b51 P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 6 to 7, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.304656 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet 149b0bffbe8a4a25afafd0f820f51b51 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:25.304919 20094 tablet_replica.cc:330] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:25.305117 20094 raft_consensus.cc:2229] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:25.305369 20094 raft_consensus.cc:2258] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:25.308013 20094 ts_tablet_manager.cc:1875] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:25.310325 20094 ts_tablet_manager.cc:1888] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:00:25.310416 20094 log.cc:1196] T 149b0bffbe8a4a25afafd0f820f51b51 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/149b0bffbe8a4a25afafd0f820f51b51
I20240503 14:00:25.310962 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet 149b0bffbe8a4a25afafd0f820f51b51 (table rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]) successfully deleted
I20240503 14:00:25.340809 19960 raft_consensus.cc:1050] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:25.341217 19960 consensus_queue.cc:237] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.342727 19161 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.342793 18154 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.343119 18286 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.343175 20115 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.343518 20051 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.343693 20067 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.345623 20115 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.346125 18286 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.346179 19161 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.349184 18007 catalog_manager.cc:5484] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.350915 18154 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.532272 18286 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:25.533676 18419 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.533751 19029 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.534119 20085 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.534341 20086 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.536286 19932 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:25.536796 19029 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:25.536801 18419 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:25.539273 18005 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:25.542732 18286 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:25.544150 19029 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:00:25.545039 18221 consensus_peers.cc:485] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc3cae9382254940bc7bb1388d05bd82. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:25.545433 20086 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:25.545594 18419 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:25.546072 20086 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:25.547392 20085 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:25.547753 18419 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.547857 19029 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:25.548264 18418 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:25.549155 19161 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.549561 20051 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.550515 17997 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bc3cae9382254940bc7bb1388d05bd82 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:25.550976 18005 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:25.552173 18154 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.552109 20051 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.552711 19161 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.553499 20067 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.555171 18154 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.555178 18013 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:25.559805 18419 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:25.561285 18154 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:25.561295 19161 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:25.561734 20051 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:25.561903 19960 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:00:25.562119 18353 consensus_peers.cc:485] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 -> Peer 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471): Couldn't send request to peer 2911b16ff0c74be48ae3a6eaa3db27cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc81b3bd2aed42f69ae05df22854aaab. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:25.563969 20067 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:25.564257 19161 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:25.566694 17998 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dc81b3bd2aed42f69ae05df22854aaab with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:25.566701 18005 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:25.567175 18154 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:25.585145 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57914
I20240503 14:00:25.585364 18419 raft_consensus.cc:595] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:25.810552 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57954
I20240503 14:00:25.810720 18419 raft_consensus.cc:595] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:25.819589 20051 raft_consensus.cc:979] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: : Instructing follower aaa289c674d34661bf49dd8bd7186a15 to start an election
I20240503 14:00:25.819711 20051 raft_consensus.cc:1067] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Signalling peer aaa289c674d34661bf49dd8bd7186a15 to start an election
I20240503 14:00:25.820436 18286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190"
dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
from {username='slave'} at 127.0.99.131:57277
I20240503 14:00:25.820631 18286 raft_consensus.cc:482] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:25.820706 18286 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:25.821599 18286 raft_consensus.cc:504] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.822100 18286 leader_election.cc:288] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 2 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181), 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929)
I20240503 14:00:25.822479 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:25.822575 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:25.822631 18154 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:25.822695 18419 raft_consensus.cc:3032] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:25.822733 18419 raft_consensus.cc:729] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:25.822671 19161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681d1aa0e244296a1457ea6faa48190" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "344bdf13092a447da55bd7a037da918a"
I20240503 14:00:25.822880 19161 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:25.822930 18419 consensus_queue.cc:260] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.823118 18419 raft_consensus.cc:3037] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:25.823676 19161 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 2.
I20240503 14:00:25.823694 18154 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 2.
I20240503 14:00:25.824136 18419 raft_consensus.cc:2454] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 2.
I20240503 14:00:25.824461 18221 leader_election.cc:302] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 344bdf13092a447da55bd7a037da918a, a9ddc910d74d4e2eb9ef49440c1fb6ad, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:25.824648 20086 raft_consensus.cc:2781] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:25.824748 20086 raft_consensus.cc:686] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 2 LEADER]: Becoming Leader. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Running, Role: LEADER
I20240503 14:00:25.824921 20086 consensus_queue.cc:237] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:25.826686 18007 catalog_manager.cc:5484] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: term changed from 1 to 2, leader changed from 625639f1ddfa446abaa3962865411504 (127.0.99.131) to aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:25.966500 20176 ts_tablet_manager.cc:902] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:25.967010 20176 tablet_copy_client.cc:308] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36605
I20240503 14:00:25.967561 18307 tablet_copy_service.cc:140] P aaa289c674d34661bf49dd8bd7186a15: Received BeginTabletCopySession request for tablet bc3cae9382254940bc7bb1388d05bd82 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:58409)
I20240503 14:00:25.967693 18307 tablet_copy_service.cc:161] P aaa289c674d34661bf49dd8bd7186a15: Beginning new tablet copy session on tablet bc3cae9382254940bc7bb1388d05bd82 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:58409: session id = 344bdf13092a447da55bd7a037da918a-bc3cae9382254940bc7bb1388d05bd82
I20240503 14:00:25.968523 18307 tablet_copy_source_session.cc:205] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:25.969520 20176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3cae9382254940bc7bb1388d05bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:25.971572 20176 tablet_copy_client.cc:783] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:25.971763 20176 tablet_copy_client.cc:647] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:25.973098 20176 tablet_copy_client.cc:523] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:25.974381 20176 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:25.984100 20182 ts_tablet_manager.cc:902] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: Initiating tablet copy from peer 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:25.984643 20182 tablet_copy_client.cc:308] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:33181
I20240503 14:00:25.985237 18439 tablet_copy_service.cc:140] P 625639f1ddfa446abaa3962865411504: Received BeginTabletCopySession request for tablet dc81b3bd2aed42f69ae05df22854aaab from peer 2911b16ff0c74be48ae3a6eaa3db27cd ({username='slave'} at 127.0.99.132:35733)
I20240503 14:00:25.985360 18439 tablet_copy_service.cc:161] P 625639f1ddfa446abaa3962865411504: Beginning new tablet copy session on tablet dc81b3bd2aed42f69ae05df22854aaab from peer 2911b16ff0c74be48ae3a6eaa3db27cd at {username='slave'} at 127.0.99.132:35733: session id = 2911b16ff0c74be48ae3a6eaa3db27cd-dc81b3bd2aed42f69ae05df22854aaab
I20240503 14:00:25.986181 18439 tablet_copy_source_session.cc:205] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:25.987174 20182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc81b3bd2aed42f69ae05df22854aaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:25.989148 20182 tablet_copy_client.cc:783] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:25.989326 20182 tablet_copy_client.cc:647] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:25.990757 20182 tablet_copy_client.cc:523] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:25.991988 20182 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap starting.
I20240503 14:00:25.994975 20176 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:25.995232 20176 tablet_bootstrap.cc:492] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:25.995414 20176 ts_tablet_manager.cc:1367] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20240503 14:00:25.995862 20176 raft_consensus.cc:348] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:25.995977 20176 raft_consensus.cc:729] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:25.996073 20176 consensus_queue.cc:260] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:25.996320 20176 ts_tablet_manager.cc:1398] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:25.997068 18307 tablet_copy_service.cc:342] P aaa289c674d34661bf49dd8bd7186a15: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-bc3cae9382254940bc7bb1388d05bd82 received from {username='slave'} at 127.0.99.133:58409
I20240503 14:00:25.997148 18307 tablet_copy_service.cc:433] P aaa289c674d34661bf49dd8bd7186a15: ending tablet copy session 344bdf13092a447da55bd7a037da918a-bc3cae9382254940bc7bb1388d05bd82 on tablet bc3cae9382254940bc7bb1388d05bd82 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:26.016000 20182 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:26.016350 20182 tablet_bootstrap.cc:492] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap complete.
I20240503 14:00:26.016564 20182 ts_tablet_manager.cc:1367] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 14:00:26.017163 20182 raft_consensus.cc:348] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:26.017293 20182 raft_consensus.cc:729] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2911b16ff0c74be48ae3a6eaa3db27cd, State: Initialized, Role: LEARNER
I20240503 14:00:26.017405 20182 consensus_queue.cc:260] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:26.017747 20182 ts_tablet_manager.cc:1398] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:26.018548 18439 tablet_copy_service.cc:342] P 625639f1ddfa446abaa3962865411504: Request end of tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-dc81b3bd2aed42f69ae05df22854aaab received from {username='slave'} at 127.0.99.132:35733
I20240503 14:00:26.018652 18439 tablet_copy_service.cc:433] P 625639f1ddfa446abaa3962865411504: ending tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-dc81b3bd2aed42f69ae05df22854aaab on tablet dc81b3bd2aed42f69ae05df22854aaab with peer 2911b16ff0c74be48ae3a6eaa3db27cd
I20240503 14:00:26.290403 18419 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:26.290855 20086 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:26.300100 18154 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:26.300698 20086 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:26.303649 19161 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:26.304714 20219 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:26.488667 19161 raft_consensus.cc:1203] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:26.542171 19029 raft_consensus.cc:1203] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:26.747215 18286 consensus_queue.cc:237] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:26.748536 19161 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:26.748620 18154 raft_consensus.cc:1261] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:26.749025 20086 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:26.749248 20219 consensus_queue.cc:1035] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:26.750906 20083 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.751380 18154 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.751413 19161 raft_consensus.cc:2932] T 1681d1aa0e244296a1457ea6faa48190 P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.753515 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1681d1aa0e244296a1457ea6faa48190 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:26.753769 18007 catalog_manager.cc:5484] T 1681d1aa0e244296a1457ea6faa48190 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 6 to 8, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:26.757004 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet 1681d1aa0e244296a1457ea6faa48190 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:26.757382 20260 tablet_replica.cc:330] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:26.757513 20260 raft_consensus.cc:2229] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.757606 20260 raft_consensus.cc:2258] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.758250 20260 ts_tablet_manager.cc:1875] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.760404 20260 ts_tablet_manager.cc:1888] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:00:26.760494 20260 log.cc:1196] T 1681d1aa0e244296a1457ea6faa48190 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1681d1aa0e244296a1457ea6faa48190
I20240503 14:00:26.761094 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet 1681d1aa0e244296a1457ea6faa48190 (table rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]) successfully deleted
I20240503 14:00:26.949545 20219 raft_consensus.cc:1050] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:26.950059 20219 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:26.951614 19161 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:26.951612 19029 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:26.951679 18419 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:26.952062 20277 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:26.952309 20219 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:26.952482 20086 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:26.954583 20219 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.955312 18419 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.955312 19161 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.957034 19029 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.957369 18005 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:26.962543 18286 consensus_queue.cc:237] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:26.963877 19161 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:26.963903 19029 raft_consensus.cc:1261] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:26.964314 20083 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:26.964545 20086 consensus_queue.cc:1035] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:26.966341 20219 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.967976 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc3cae9382254940bc7bb1388d05bd82 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:26.968083 19029 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.968477 19161 raft_consensus.cc:2932] T bc3cae9382254940bc7bb1388d05bd82 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:26.969185 18013 catalog_manager.cc:5484] T bc3cae9382254940bc7bb1388d05bd82 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:26.972054 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3cae9382254940bc7bb1388d05bd82 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:26.972323 20260 tablet_replica.cc:330] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:26.972446 20260 raft_consensus.cc:2229] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.972673 20260 raft_consensus.cc:2258] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.973654 20260 ts_tablet_manager.cc:1875] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.975903 20260 ts_tablet_manager.cc:1888] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:00:26.975983 20260 log.cc:1196] T bc3cae9382254940bc7bb1388d05bd82 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc3cae9382254940bc7bb1388d05bd82
I20240503 14:00:26.976495 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet bc3cae9382254940bc7bb1388d05bd82 (table rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]) successfully deleted
I20240503 14:00:27.094512 20051 raft_consensus.cc:1050] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: attempting to promote NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd to VOTER
I20240503 14:00:27.094936 20051 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:27.096450 19029 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:27.096895 18154 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:27.096906 20281 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:27.097280 20281 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:27.097462 19161 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:27.098326 20281 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:27.099579 20051 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.100919 18154 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.101315 19161 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.102388 19029 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.102609 18007 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:27.120301 18419 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:27.121770 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:27.121791 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:27.122304 20284 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.122531 20281 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.124501 20284 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.125000 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.125316 19029 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:27.127593 18010 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:27.131492 18419 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:27.133033 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.133204 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.133554 20283 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:00:27.133785 18353 consensus_peers.cc:485] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 -> Peer 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): Couldn't send request to peer 344bdf13092a447da55bd7a037da918a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae4aca78536e4b97bdf12e0c41ea7614. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.133960 20283 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.135366 20281 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:27.135936 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:27.136548 19029 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } } }
I20240503 14:00:27.137394 17998 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae4aca78536e4b97bdf12e0c41ea7614 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:27.138648 18007 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 7 to 8, NON_VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:27.151552 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58008
I20240503 14:00:27.151716 18419 raft_consensus.cc:595] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:27.366329 18154 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:27.367619 19161 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:27.367695 18419 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:27.368055 20323 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.368300 19568 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.370287 19568 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:27.370672 19161 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:27.370704 18419 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:27.373288 18010 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:27.376385 18154 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:27.378063 19161 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.378096 18419 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.378430 20326 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.378654 19568 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.380393 20323 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:27.381263 18419 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:27.382169 19161 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:27.383698 18005 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:00:27.384230 18089 consensus_peers.cc:485] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad -> Peer 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471): Couldn't send request to peer 2911b16ff0c74be48ae3a6eaa3db27cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a163e1ed7c02469fb7c87e4953f0e517. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.384532 17998 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a163e1ed7c02469fb7c87e4953f0e517 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:27.400713 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58036
I20240503 14:00:27.400877 18419 raft_consensus.cc:595] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:27.535928 20283 raft_consensus.cc:979] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: : Instructing follower 344bdf13092a447da55bd7a037da918a to start an election
I20240503 14:00:27.536072 20283 raft_consensus.cc:1067] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Signalling peer 344bdf13092a447da55bd7a037da918a to start an election
I20240503 14:00:27.536624 19161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab"
dest_uuid: "344bdf13092a447da55bd7a037da918a"
from {username='slave'} at 127.0.99.131:60227
I20240503 14:00:27.536842 19161 raft_consensus.cc:482] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:27.536911 19161 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.537760 19161 raft_consensus.cc:504] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:27.538921 19161 leader_election.cc:288] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [CANDIDATE]: Term 2 election: Requested vote from peers a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181), 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471)
I20240503 14:00:27.539319 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "344bdf13092a447da55bd7a037da918a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad"
I20240503 14:00:27.539402 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "344bdf13092a447da55bd7a037da918a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:27.539469 18154 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.539549 18419 raft_consensus.cc:3032] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:27.539592 18419 raft_consensus.cc:729] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:27.539752 18419 consensus_queue.cc:260] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:27.539968 18419 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.540506 18154 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 344bdf13092a447da55bd7a037da918a in term 2.
I20240503 14:00:27.541006 18419 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 344bdf13092a447da55bd7a037da918a in term 2.
I20240503 14:00:27.541337 19095 leader_election.cc:302] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 344bdf13092a447da55bd7a037da918a, 625639f1ddfa446abaa3962865411504, a9ddc910d74d4e2eb9ef49440c1fb6ad; no voters:
I20240503 14:00:27.541594 20223 raft_consensus.cc:2781] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:27.541841 20223 raft_consensus.cc:686] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 2 LEADER]: Becoming Leader. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Running, Role: LEADER
I20240503 14:00:27.542210 20223 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:27.543641 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc81b3bd2aed42f69ae05df22854aaab" candidate_uuid: "344bdf13092a447da55bd7a037da918a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
I20240503 14:00:27.543815 19029 raft_consensus.cc:3037] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.544912 19029 raft_consensus.cc:2454] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 344bdf13092a447da55bd7a037da918a in term 2.
I20240503 14:00:27.544950 18010 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a reported cstate change: term changed from 1 to 2, leader changed from 625639f1ddfa446abaa3962865411504 (127.0.99.131) to 344bdf13092a447da55bd7a037da918a (127.0.99.133). New cstate: current_term: 2 leader_uuid: "344bdf13092a447da55bd7a037da918a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:27.670475 20367 ts_tablet_manager.cc:902] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: Initiating tablet copy from peer 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181)
I20240503 14:00:27.670993 20367 tablet_copy_client.cc:308] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:33181
I20240503 14:00:27.671586 18439 tablet_copy_service.cc:140] P 625639f1ddfa446abaa3962865411504: Received BeginTabletCopySession request for tablet ae4aca78536e4b97bdf12e0c41ea7614 from peer 344bdf13092a447da55bd7a037da918a ({username='slave'} at 127.0.99.133:57245)
I20240503 14:00:27.671705 18439 tablet_copy_service.cc:161] P 625639f1ddfa446abaa3962865411504: Beginning new tablet copy session on tablet ae4aca78536e4b97bdf12e0c41ea7614 from peer 344bdf13092a447da55bd7a037da918a at {username='slave'} at 127.0.99.133:57245: session id = 344bdf13092a447da55bd7a037da918a-ae4aca78536e4b97bdf12e0c41ea7614
I20240503 14:00:27.672452 18439 tablet_copy_source_session.cc:205] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.673493 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4aca78536e4b97bdf12e0c41ea7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.675916 20367 tablet_copy_client.cc:783] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.676158 20367 tablet_copy_client.cc:647] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.677371 20367 tablet_copy_client.cc:523] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.678599 20367 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: Bootstrap starting.
I20240503 14:00:27.697942 20367 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.698338 20367 tablet_bootstrap.cc:492] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: Bootstrap complete.
I20240503 14:00:27.698542 20367 ts_tablet_manager.cc:1367] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.007s
I20240503 14:00:27.699021 20367 raft_consensus.cc:348] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:27.699151 20367 raft_consensus.cc:729] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Initialized, Role: LEARNER
I20240503 14:00:27.699260 20367 consensus_queue.cc:260] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: true } }
I20240503 14:00:27.699532 20367 ts_tablet_manager.cc:1398] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:27.700215 18439 tablet_copy_service.cc:342] P 625639f1ddfa446abaa3962865411504: Request end of tablet copy session 344bdf13092a447da55bd7a037da918a-ae4aca78536e4b97bdf12e0c41ea7614 received from {username='slave'} at 127.0.99.133:57245
I20240503 14:00:27.700320 18439 tablet_copy_service.cc:433] P 625639f1ddfa446abaa3962865411504: ending tablet copy session 344bdf13092a447da55bd7a037da918a-ae4aca78536e4b97bdf12e0c41ea7614 on tablet ae4aca78536e4b97bdf12e0c41ea7614 with peer 344bdf13092a447da55bd7a037da918a
I20240503 14:00:27.949810 19029 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:27.950428 20223 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:27.954938 20389 ts_tablet_manager.cc:902] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: Initiating tablet copy from peer a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:27.955495 20389 tablet_copy_client.cc:308] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33467
I20240503 14:00:27.959051 18175 tablet_copy_service.cc:140] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Received BeginTabletCopySession request for tablet a163e1ed7c02469fb7c87e4953f0e517 from peer 2911b16ff0c74be48ae3a6eaa3db27cd ({username='slave'} at 127.0.99.132:35443)
I20240503 14:00:27.959172 18175 tablet_copy_service.cc:161] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Beginning new tablet copy session on tablet a163e1ed7c02469fb7c87e4953f0e517 from peer 2911b16ff0c74be48ae3a6eaa3db27cd at {username='slave'} at 127.0.99.132:35443: session id = 2911b16ff0c74be48ae3a6eaa3db27cd-a163e1ed7c02469fb7c87e4953f0e517
I20240503 14:00:27.960024 18175 tablet_copy_source_session.cc:205] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.960966 20389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a163e1ed7c02469fb7c87e4953f0e517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.963092 20389 tablet_copy_client.cc:783] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.963268 20389 tablet_copy_client.cc:647] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.964563 20389 tablet_copy_client.cc:523] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.965771 20389 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap starting.
I20240503 14:00:27.966524 18419 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:27.967017 20223 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:27.970268 18154 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:27.970655 20223 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:27.988026 20389 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.988334 20389 tablet_bootstrap.cc:492] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap complete.
I20240503 14:00:27.988543 20389 ts_tablet_manager.cc:1367] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.001s
I20240503 14:00:27.988978 20389 raft_consensus.cc:348] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:27.989099 20389 raft_consensus.cc:729] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2911b16ff0c74be48ae3a6eaa3db27cd, State: Initialized, Role: LEARNER
I20240503 14:00:27.989192 20389 consensus_queue.cc:260] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:27.989459 20389 ts_tablet_manager.cc:1398] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:27.990105 18175 tablet_copy_service.cc:342] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Request end of tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-a163e1ed7c02469fb7c87e4953f0e517 received from {username='slave'} at 127.0.99.132:35443
I20240503 14:00:27.990218 18175 tablet_copy_service.cc:433] P a9ddc910d74d4e2eb9ef49440c1fb6ad: ending tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-a163e1ed7c02469fb7c87e4953f0e517 on tablet a163e1ed7c02469fb7c87e4953f0e517 with peer 2911b16ff0c74be48ae3a6eaa3db27cd
I20240503 14:00:28.224920 19161 raft_consensus.cc:1203] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:28.462385 19029 raft_consensus.cc:1203] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:28.543202 19161 consensus_queue.cc:237] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:28.544495 19029 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:28.544500 18154 raft_consensus.cc:1261] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:28.544941 20397 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:28.545186 20450 consensus_queue.cc:1035] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:28.547326 20452 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 344bdf13092a447da55bd7a037da918a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.547479 18154 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.547469 19029 raft_consensus.cc:2932] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.549728 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc81b3bd2aed42f69ae05df22854aaab with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:28.550495 18007 catalog_manager.cc:5484] T dc81b3bd2aed42f69ae05df22854aaab P 2911b16ff0c74be48ae3a6eaa3db27cd reported cstate change: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "344bdf13092a447da55bd7a037da918a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.553536 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet dc81b3bd2aed42f69ae05df22854aaab with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:28.553990 20457 tablet_replica.cc:330] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:28.554139 20457 raft_consensus.cc:2229] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:28.554267 20457 raft_consensus.cc:2258] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:28.554885 20457 ts_tablet_manager.cc:1875] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:28.556862 20457 ts_tablet_manager.cc:1888] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:00:28.556934 20457 log.cc:1196] T dc81b3bd2aed42f69ae05df22854aaab P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc81b3bd2aed42f69ae05df22854aaab
I20240503 14:00:28.557472 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet dc81b3bd2aed42f69ae05df22854aaab (table rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]) successfully deleted
I20240503 14:00:28.671583 20418 raft_consensus.cc:1050] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: attempting to promote NON_VOTER 344bdf13092a447da55bd7a037da918a to VOTER
I20240503 14:00:28.672013 20418 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:28.673543 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.673543 19161 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 LEARNER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.673543 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.673964 20459 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.674281 20417 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.674489 20418 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.676493 20418 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:28.677012 19029 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:28.677012 19161 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:28.678531 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:28.679467 18010 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 8 to 9, 344bdf13092a447da55bd7a037da918a (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:28.725333 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58108
I20240503 14:00:28.725530 18419 raft_consensus.cc:595] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:28.941910 18419 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:28.943632 18286 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:28.943639 19029 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:28.944126 20283 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:28.944548 20417 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:28.946413 20417 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.949283 18005 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:28.951606 19029 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.951957 18286 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:28.952868 18419 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } }
W20240503 14:00:28.956643 18353 consensus_peers.cc:485] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 -> Peer a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467): Couldn't send request to peer a9ddc910d74d4e2eb9ef49440c1fb6ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.957262 18286 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:28.958411 19029 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer 625639f1ddfa446abaa3962865411504: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:28.958456 20283 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.959065 20283 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.960294 20459 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } } }
I20240503 14:00:28.962373 17998 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bb2ca36226704129a6522e683e6b9dda with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:28.963106 19029 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } } }
I20240503 14:00:28.963392 18013 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 7 to 8, NON_VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "625639f1ddfa446abaa3962865411504" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:28.964517 18286 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } } }
I20240503 14:00:28.971300 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58140
I20240503 14:00:28.971469 18419 raft_consensus.cc:595] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:28.989293 18419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614"
dest_uuid: "625639f1ddfa446abaa3962865411504"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58158
I20240503 14:00:28.989475 18419 raft_consensus.cc:595] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:28.998870 20434 raft_consensus.cc:1050] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad: attempting to promote NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd to VOTER
I20240503 14:00:28.999298 20434 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:29.000725 19029 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:29.000814 18419 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:29.000828 19161 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:29.001183 20516 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.001384 20518 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.001617 20519 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.003600 20516 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.004096 19161 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.004437 19029 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.006521 18007 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad reported cstate change: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:29.006831 18419 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.009797 18154 consensus_queue.cc:237] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:29.011128 19029 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:29.011138 19161 raft_consensus.cc:1261] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:29.011581 20516 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:29.011801 20518 consensus_queue.cc:1035] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:29.013363 20434 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.013831 19029 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.013928 19161 raft_consensus.cc:2932] T a163e1ed7c02469fb7c87e4953f0e517 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.015246 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a163e1ed7c02469fb7c87e4953f0e517 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:29.016260 18005 catalog_manager.cc:5484] T a163e1ed7c02469fb7c87e4953f0e517 P 2911b16ff0c74be48ae3a6eaa3db27cd reported cstate change: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:29.018774 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet a163e1ed7c02469fb7c87e4953f0e517 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:29.019016 20457 tablet_replica.cc:330] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:29.019160 20457 raft_consensus.cc:2229] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:29.019383 20457 raft_consensus.cc:2258] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:29.020155 20457 ts_tablet_manager.cc:1875] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:29.022143 20457 ts_tablet_manager.cc:1888] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:00:29.022224 20457 log.cc:1196] T a163e1ed7c02469fb7c87e4953f0e517 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a163e1ed7c02469fb7c87e4953f0e517
I20240503 14:00:29.022768 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet a163e1ed7c02469fb7c87e4953f0e517 (table rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]) successfully deleted
I20240503 14:00:29.180128 20283 raft_consensus.cc:979] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: : Instructing follower 344bdf13092a447da55bd7a037da918a to start an election
I20240503 14:00:29.180251 20283 raft_consensus.cc:1067] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Signalling peer 344bdf13092a447da55bd7a037da918a to start an election
I20240503 14:00:29.180804 19161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614"
dest_uuid: "344bdf13092a447da55bd7a037da918a"
from {username='slave'} at 127.0.99.131:60227
I20240503 14:00:29.180981 19161 raft_consensus.cc:482] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:29.181056 19161 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.181967 19161 raft_consensus.cc:504] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:29.182603 19161 leader_election.cc:288] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [CANDIDATE]: Term 2 election: Requested vote from peers aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605), 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181), 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471)
I20240503 14:00:29.183015 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "344bdf13092a447da55bd7a037da918a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:29.183019 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "344bdf13092a447da55bd7a037da918a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
I20240503 14:00:29.183064 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4aca78536e4b97bdf12e0c41ea7614" candidate_uuid: "344bdf13092a447da55bd7a037da918a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
I20240503 14:00:29.183167 18286 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.183176 19029 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.183166 18419 raft_consensus.cc:3032] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:29.183218 18419 raft_consensus.cc:729] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:29.183379 18419 consensus_queue.cc:260] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:29.183611 18419 raft_consensus.cc:3037] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.184298 19029 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 344bdf13092a447da55bd7a037da918a in term 2.
I20240503 14:00:29.184304 18286 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 344bdf13092a447da55bd7a037da918a in term 2.
I20240503 14:00:29.184475 18419 raft_consensus.cc:2454] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 344bdf13092a447da55bd7a037da918a in term 2.
I20240503 14:00:29.184764 19095 leader_election.cc:302] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2911b16ff0c74be48ae3a6eaa3db27cd, 344bdf13092a447da55bd7a037da918a, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:29.184994 20450 raft_consensus.cc:2781] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:29.185171 20450 raft_consensus.cc:686] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 2 LEADER]: Becoming Leader. State: Replica: 344bdf13092a447da55bd7a037da918a, State: Running, Role: LEADER
I20240503 14:00:29.185356 20450 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:29.187265 18007 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a reported cstate change: term changed from 1 to 2, leader changed from 625639f1ddfa446abaa3962865411504 (127.0.99.131) to 344bdf13092a447da55bd7a037da918a (127.0.99.133). New cstate: current_term: 2 leader_uuid: "344bdf13092a447da55bd7a037da918a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:29.358412 20459 raft_consensus.cc:979] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: : Instructing follower aaa289c674d34661bf49dd8bd7186a15 to start an election
I20240503 14:00:29.358548 20459 raft_consensus.cc:1067] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Signalling peer aaa289c674d34661bf49dd8bd7186a15 to start an election
I20240503 14:00:29.359143 18286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda"
dest_uuid: "aaa289c674d34661bf49dd8bd7186a15"
from {username='slave'} at 127.0.99.131:57277
I20240503 14:00:29.359328 18286 raft_consensus.cc:482] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:29.359400 18286 raft_consensus.cc:3037] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.360342 18286 raft_consensus.cc:504] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } }
I20240503 14:00:29.360746 18286 leader_election.cc:288] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 2 election: Requested vote from peers 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181), 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471)
I20240503 14:00:29.361286 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "625639f1ddfa446abaa3962865411504"
I20240503 14:00:29.361280 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca36226704129a6522e683e6b9dda" candidate_uuid: "aaa289c674d34661bf49dd8bd7186a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd"
I20240503 14:00:29.361457 18419 raft_consensus.cc:3032] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:29.361466 19029 raft_consensus.cc:3037] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.361502 18419 raft_consensus.cc:729] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 625639f1ddfa446abaa3962865411504, State: Running, Role: LEADER
I20240503 14:00:29.361703 18419 consensus_queue.cc:260] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } }
I20240503 14:00:29.361948 18419 raft_consensus.cc:3037] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:29.362620 19029 raft_consensus.cc:2454] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 2.
I20240503 14:00:29.362972 18221 leader_election.cc:302] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2911b16ff0c74be48ae3a6eaa3db27cd, aaa289c674d34661bf49dd8bd7186a15; no voters:
I20240503 14:00:29.363018 18419 raft_consensus.cc:2454] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa289c674d34661bf49dd8bd7186a15 in term 2.
I20240503 14:00:29.363193 20539 raft_consensus.cc:2781] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:29.363436 20539 raft_consensus.cc:686] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 2 LEADER]: Becoming Leader. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Running, Role: LEADER
I20240503 14:00:29.363619 20539 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } }
I20240503 14:00:29.365657 18007 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: term changed from 1 to 2, leader changed from 625639f1ddfa446abaa3962865411504 (127.0.99.131) to aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:29.419097 18154 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:29.420522 18419 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:29.420550 19161 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:29.420967 20518 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:29.421198 20519 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:29.423328 20516 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:29.423928 19161 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:29.425784 18419 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:29.426822 18007 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:29.431299 18154 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } }
I20240503 14:00:29.433014 19161 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:29.433020 18419 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:00:29.433409 18088 consensus_peers.cc:485] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad -> Peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605): Couldn't send request to peer aaa289c674d34661bf49dd8bd7186a15. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.433517 20516 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:29.434124 20518 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:29.435773 20519 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } } }
I20240503 14:00:29.436141 18419 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } } }
I20240503 14:00:29.436347 19161 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } } }
I20240503 14:00:29.437924 17998 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 30e32918970a4daa9d27134dfaa65a24 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:00:29.438866 18010 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 reported cstate change: config changed from index 7 to 8, NON_VOTER aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } } }
I20240503 14:00:29.621806 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:29.622359 20450 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 14:00:29.749783 18221 consensus_peers.cc:485] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 -> Peer a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467): Couldn't send request to peer a9ddc910d74d4e2eb9ef49440c1fb6ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.781611 19029 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:29.782299 20539 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.785089 18419 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:29.785547 20086 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.789182 18419 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:29.789647 20450 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:29.791286 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:29.791906 20585 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:29.939463 20604 ts_tablet_manager.cc:902] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Initiating tablet copy from peer a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129:33467)
I20240503 14:00:29.940346 20604 tablet_copy_client.cc:308] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33467
I20240503 14:00:29.941187 18175 tablet_copy_service.cc:140] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Received BeginTabletCopySession request for tablet 30e32918970a4daa9d27134dfaa65a24 from peer aaa289c674d34661bf49dd8bd7186a15 ({username='slave'} at 127.0.99.130:40075)
I20240503 14:00:29.941319 18175 tablet_copy_service.cc:161] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Beginning new tablet copy session on tablet 30e32918970a4daa9d27134dfaa65a24 from peer aaa289c674d34661bf49dd8bd7186a15 at {username='slave'} at 127.0.99.130:40075: session id = aaa289c674d34661bf49dd8bd7186a15-30e32918970a4daa9d27134dfaa65a24
I20240503 14:00:29.942123 18175 tablet_copy_source_session.cc:205] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.943284 20604 ts_tablet_manager.cc:1875] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:29.945703 20604 ts_tablet_manager.cc:1888] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:00:29.945915 20604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e32918970a4daa9d27134dfaa65a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.947117 20604 tablet_copy_client.cc:783] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.947299 20604 tablet_copy_client.cc:647] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.949043 20604 tablet_copy_client.cc:523] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.950624 20604 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap starting.
I20240503 14:00:29.972314 20604 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.972681 20604 tablet_bootstrap.cc:492] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Bootstrap complete.
I20240503 14:00:29.972883 20604 ts_tablet_manager.cc:1367] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240503 14:00:29.973384 20604 raft_consensus.cc:348] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } }
I20240503 14:00:29.973510 20604 raft_consensus.cc:729] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aaa289c674d34661bf49dd8bd7186a15, State: Initialized, Role: LEARNER
I20240503 14:00:29.973615 20604 consensus_queue.cc:260] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: true } }
I20240503 14:00:29.973915 20604 ts_tablet_manager.cc:1398] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:29.974742 18175 tablet_copy_service.cc:342] P a9ddc910d74d4e2eb9ef49440c1fb6ad: Request end of tablet copy session aaa289c674d34661bf49dd8bd7186a15-30e32918970a4daa9d27134dfaa65a24 received from {username='slave'} at 127.0.99.130:40075
I20240503 14:00:29.974831 18175 tablet_copy_service.cc:433] P a9ddc910d74d4e2eb9ef49440c1fb6ad: ending tablet copy session aaa289c674d34661bf49dd8bd7186a15-30e32918970a4daa9d27134dfaa65a24 on tablet 30e32918970a4daa9d27134dfaa65a24 with peer aaa289c674d34661bf49dd8bd7186a15
I20240503 14:00:30.261353 19161 consensus_queue.cc:237] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } }
I20240503 14:00:30.263402 19029 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:30.263826 18286 raft_consensus.cc:1261] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Refusing update from remote peer 344bdf13092a447da55bd7a037da918a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:30.263938 20585 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:30.264240 20586 consensus_queue.cc:1035] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:30.266149 20450 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 344bdf13092a447da55bd7a037da918a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:30.266628 19029 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:30.266712 18286 raft_consensus.cc:2932] T ae4aca78536e4b97bdf12e0c41ea7614 P aaa289c674d34661bf49dd8bd7186a15 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:30.268287 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae4aca78536e4b97bdf12e0c41ea7614 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:30.269591 18007 catalog_manager.cc:5484] T ae4aca78536e4b97bdf12e0c41ea7614 P 2911b16ff0c74be48ae3a6eaa3db27cd reported cstate change: config changed from index 9 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "344bdf13092a447da55bd7a037da918a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } }
I20240503 14:00:30.272881 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4aca78536e4b97bdf12e0c41ea7614 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:30.273515 20628 tablet_replica.cc:330] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:30.273654 20628 raft_consensus.cc:2229] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:30.273772 20628 raft_consensus.cc:2258] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:30.274590 20628 ts_tablet_manager.cc:1875] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:30.276789 20628 ts_tablet_manager.cc:1888] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:00:30.276885 20628 log.cc:1196] T ae4aca78536e4b97bdf12e0c41ea7614 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae4aca78536e4b97bdf12e0c41ea7614
I20240503 14:00:30.277540 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet ae4aca78536e4b97bdf12e0c41ea7614 (table rebalance_test_table_1 [id=a3c24120807548d486b743ca583abb00]) successfully deleted
I20240503 14:00:30.332412 20644 ts_tablet_manager.cc:902] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Initiating tablet copy from peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:30.334475 20644 tablet_copy_client.cc:308] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36605
I20240503 14:00:30.335295 18307 tablet_copy_service.cc:140] P aaa289c674d34661bf49dd8bd7186a15: Received BeginTabletCopySession request for tablet bb2ca36226704129a6522e683e6b9dda from peer a9ddc910d74d4e2eb9ef49440c1fb6ad ({username='slave'} at 127.0.99.129:48609)
I20240503 14:00:30.335419 18307 tablet_copy_service.cc:161] P aaa289c674d34661bf49dd8bd7186a15: Beginning new tablet copy session on tablet bb2ca36226704129a6522e683e6b9dda from peer a9ddc910d74d4e2eb9ef49440c1fb6ad at {username='slave'} at 127.0.99.129:48609: session id = a9ddc910d74d4e2eb9ef49440c1fb6ad-bb2ca36226704129a6522e683e6b9dda
I20240503 14:00:30.336313 18307 tablet_copy_source_session.cc:205] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:30.337967 20644 ts_tablet_manager.cc:1875] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:30.340040 20644 ts_tablet_manager.cc:1888] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:00:30.340234 20644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca36226704129a6522e683e6b9dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:30.341367 20644 tablet_copy_client.cc:783] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:30.341549 20644 tablet_copy_client.cc:647] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:30.343501 20644 tablet_copy_client.cc:523] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:30.345080 20644 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap starting.
I20240503 14:00:30.367355 20644 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:30.367718 20644 tablet_bootstrap.cc:492] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Bootstrap complete.
I20240503 14:00:30.367935 20644 ts_tablet_manager.cc:1367] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240503 14:00:30.368505 20644 raft_consensus.cc:348] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } }
I20240503 14:00:30.368690 20644 raft_consensus.cc:729] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a9ddc910d74d4e2eb9ef49440c1fb6ad, State: Initialized, Role: LEARNER
I20240503 14:00:30.368820 20644 consensus_queue.cc:260] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: true } }
I20240503 14:00:30.369534 20644 ts_tablet_manager.cc:1398] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:30.370177 18307 tablet_copy_service.cc:342] P aaa289c674d34661bf49dd8bd7186a15: Request end of tablet copy session a9ddc910d74d4e2eb9ef49440c1fb6ad-bb2ca36226704129a6522e683e6b9dda received from {username='slave'} at 127.0.99.129:48609
I20240503 14:00:30.370280 18307 tablet_copy_service.cc:433] P aaa289c674d34661bf49dd8bd7186a15: ending tablet copy session a9ddc910d74d4e2eb9ef49440c1fb6ad-bb2ca36226704129a6522e683e6b9dda on tablet bb2ca36226704129a6522e683e6b9dda with peer a9ddc910d74d4e2eb9ef49440c1fb6ad
I20240503 14:00:30.529197 18286 raft_consensus.cc:1203] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:30.878981 18154 raft_consensus.cc:1203] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 LEARNER]: Deduplicated request from leader. Original: 1.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 14:00:31.093042 20684 raft_consensus.cc:1050] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad: attempting to promote NON_VOTER aaa289c674d34661bf49dd8bd7186a15 to VOTER
I20240503 14:00:31.093480 20684 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } }
I20240503 14:00:31.095014 18286 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEARNER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:31.095038 19161 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:31.095046 18419 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:31.095466 20686 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:31.095759 20651 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:31.095992 20687 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:31.098335 20687 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.098778 18419 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.099781 18286 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.102246 19161 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.102420 18013 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 8 to 9, aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.106680 18154 consensus_queue.cc:237] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } }
I20240503 14:00:31.107993 18286 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:31.108179 19161 raft_consensus.cc:1261] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer a9ddc910d74d4e2eb9ef49440c1fb6ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:31.108502 20687 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:31.108738 20686 consensus_queue.cc:1035] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:31.110520 20651 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.111009 18286 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.111101 19161 raft_consensus.cc:2932] T 30e32918970a4daa9d27134dfaa65a24 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.112529 17998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30e32918970a4daa9d27134dfaa65a24 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:31.113546 18007 catalog_manager.cc:5484] T 30e32918970a4daa9d27134dfaa65a24 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 9 to 10, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } attrs { promote: false } } }
I20240503 14:00:31.116485 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e32918970a4daa9d27134dfaa65a24 with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:31.116963 20695 tablet_replica.cc:330] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:31.117095 20695 raft_consensus.cc:2229] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:31.117287 20695 raft_consensus.cc:2258] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:31.118060 20695 ts_tablet_manager.cc:1875] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:31.120189 20695 ts_tablet_manager.cc:1888] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:00:31.120278 20695 log.cc:1196] T 30e32918970a4daa9d27134dfaa65a24 P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/30e32918970a4daa9d27134dfaa65a24
I20240503 14:00:31.120836 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet 30e32918970a4daa9d27134dfaa65a24 (table rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]) successfully deleted
I20240503 14:00:31.376614 20672 raft_consensus.cc:1050] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15: attempting to promote NON_VOTER a9ddc910d74d4e2eb9ef49440c1fb6ad to VOTER
I20240503 14:00:31.377071 20672 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } }
I20240503 14:00:31.378672 18419 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:31.378742 19029 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:31.379091 18154 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 LEARNER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:31.379137 20707 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:31.379429 20086 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:31.379673 20707 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:31.381687 20086 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.382233 19029 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.382444 18154 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.385010 18010 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 8 to 10, a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:31.385449 18419 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, a9ddc910d74d4e2eb9ef49440c1fb6ad (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "625639f1ddfa446abaa3962865411504" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33181 } attrs { replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.389083 18286 consensus_queue.cc:237] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } }
I20240503 14:00:31.390527 18154 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:31.390527 19029 raft_consensus.cc:1261] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:31.391027 20673 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:31.391458 20673 consensus_queue.cc:1035] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:31.393289 20672 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P aaa289c674d34661bf49dd8bd7186a15 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.394043 19029 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.394277 18154 raft_consensus.cc:2932] T bb2ca36226704129a6522e683e6b9dda P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.395789 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb2ca36226704129a6522e683e6b9dda with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:31.396845 18005 catalog_manager.cc:5484] T bb2ca36226704129a6522e683e6b9dda P 2911b16ff0c74be48ae3a6eaa3db27cd reported cstate change: config changed from index 10 to 11, VOTER 625639f1ddfa446abaa3962865411504 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } attrs { promote: false } } }
I20240503 14:00:31.399870 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2ca36226704129a6522e683e6b9dda with delete_type TABLET_DATA_TOMBSTONED (TS 625639f1ddfa446abaa3962865411504 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45892
I20240503 14:00:31.400110 20695 tablet_replica.cc:330] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: stopping tablet replica
I20240503 14:00:31.400231 20695 raft_consensus.cc:2229] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:31.400401 20695 raft_consensus.cc:2258] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:31.401181 20695 ts_tablet_manager.cc:1875] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:31.403460 20695 ts_tablet_manager.cc:1888] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:00:31.403534 20695 log.cc:1196] T bb2ca36226704129a6522e683e6b9dda P 625639f1ddfa446abaa3962865411504: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb2ca36226704129a6522e683e6b9dda
I20240503 14:00:31.404011 17998 catalog_manager.cc:4830] TS 625639f1ddfa446abaa3962865411504 (127.0.99.131:33181): tablet bb2ca36226704129a6522e683e6b9dda (table rebalance_test_table_2 [id=ca37778566e14467b69bab9208109948]) successfully deleted
I20240503 14:00:31.657021 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.661126 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.662103 19141 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.664055 18397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.671095 18265 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:31.764196 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.765859 19141 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.766681 18265 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:31.767544 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.770613 18397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.868609 18397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.871184 18265 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:31.872442 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.872622 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.892665 19141 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.975581 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.977254 19141 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:31.978958 18265 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:31.979976 18397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.000903 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:32.029819 18286 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:32.031564 18154 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:32.031630 19161 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:00:32.031733 18221 consensus_peers.cc:485] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 -> Peer 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132:36471): Couldn't send request to peer 2911b16ff0c74be48ae3a6eaa3db27cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500961d6706043ca9bd3fd2eb3f274a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:32.032196 20707 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:32.032572 20859 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:32.034065 20707 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:32.034988 19161 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:32.035888 18154 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } } }
I20240503 14:00:32.037871 18013 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 7 to 8, NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:32.408674 20875 ts_tablet_manager.cc:902] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: Initiating tablet copy from peer aaa289c674d34661bf49dd8bd7186a15 (127.0.99.130:36605)
I20240503 14:00:32.409226 20875 tablet_copy_client.cc:308] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36605
I20240503 14:00:32.409889 18307 tablet_copy_service.cc:140] P aaa289c674d34661bf49dd8bd7186a15: Received BeginTabletCopySession request for tablet 500961d6706043ca9bd3fd2eb3f274a4 from peer 2911b16ff0c74be48ae3a6eaa3db27cd ({username='slave'} at 127.0.99.132:59599)
I20240503 14:00:32.410045 18307 tablet_copy_service.cc:161] P aaa289c674d34661bf49dd8bd7186a15: Beginning new tablet copy session on tablet 500961d6706043ca9bd3fd2eb3f274a4 from peer 2911b16ff0c74be48ae3a6eaa3db27cd at {username='slave'} at 127.0.99.132:59599: session id = 2911b16ff0c74be48ae3a6eaa3db27cd-500961d6706043ca9bd3fd2eb3f274a4
I20240503 14:00:32.410931 18307 tablet_copy_source_session.cc:205] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:32.411906 20875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500961d6706043ca9bd3fd2eb3f274a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:32.413822 20875 tablet_copy_client.cc:783] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:32.414012 20875 tablet_copy_client.cc:647] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:32.415446 20875 tablet_copy_client.cc:523] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:32.416913 20875 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap starting.
I20240503 14:00:32.439002 20875 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:32.439337 20875 tablet_bootstrap.cc:492] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: Bootstrap complete.
I20240503 14:00:32.439527 20875 ts_tablet_manager.cc:1367] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.001s
I20240503 14:00:32.440025 20875 raft_consensus.cc:348] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:32.440158 20875 raft_consensus.cc:729] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2911b16ff0c74be48ae3a6eaa3db27cd, State: Initialized, Role: LEARNER
I20240503 14:00:32.440258 20875 consensus_queue.cc:260] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: true } }
I20240503 14:00:32.440623 20875 ts_tablet_manager.cc:1398] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:32.441356 18307 tablet_copy_service.cc:342] P aaa289c674d34661bf49dd8bd7186a15: Request end of tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-500961d6706043ca9bd3fd2eb3f274a4 received from {username='slave'} at 127.0.99.132:59599
I20240503 14:00:32.441460 18307 tablet_copy_service.cc:433] P aaa289c674d34661bf49dd8bd7186a15: ending tablet copy session 2911b16ff0c74be48ae3a6eaa3db27cd-500961d6706043ca9bd3fd2eb3f274a4 on tablet 500961d6706043ca9bd3fd2eb3f274a4 with peer 2911b16ff0c74be48ae3a6eaa3db27cd
I20240503 14:00:32.842917 19029 raft_consensus.cc:1203] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:33.420603 20893 raft_consensus.cc:1050] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15: attempting to promote NON_VOTER 2911b16ff0c74be48ae3a6eaa3db27cd to VOTER
I20240503 14:00:33.421041 20893 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:33.422632 18155 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:33.422734 19161 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:33.422734 19030 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 LEARNER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:33.423152 20911 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:33.423401 20909 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:33.423574 20893 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:33.425649 20911 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.426331 19161 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.426594 18155 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.428246 19030 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.429374 18005 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a reported cstate change: config changed from index 8 to 9, 2911b16ff0c74be48ae3a6eaa3db27cd (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "344bdf13092a447da55bd7a037da918a" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39929 } attrs { promote: false replace: true } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.433514 18286 consensus_queue.cc:237] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } }
I20240503 14:00:33.434962 18155 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:33.435390 19030 raft_consensus.cc:1261] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Refusing update from remote peer aaa289c674d34661bf49dd8bd7186a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:33.435398 20893 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:33.435935 20893 consensus_queue.cc:1035] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:33.437242 20909 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.437919 18155 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P a9ddc910d74d4e2eb9ef49440c1fb6ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.439364 17997 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 500961d6706043ca9bd3fd2eb3f274a4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:33.439613 19030 raft_consensus.cc:2932] T 500961d6706043ca9bd3fd2eb3f274a4 P 2911b16ff0c74be48ae3a6eaa3db27cd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } } }
I20240503 14:00:33.440265 18010 catalog_manager.cc:5484] T 500961d6706043ca9bd3fd2eb3f274a4 P aaa289c674d34661bf49dd8bd7186a15 reported cstate change: config changed from index 9 to 10, VOTER 344bdf13092a447da55bd7a037da918a (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "aaa289c674d34661bf49dd8bd7186a15" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a9ddc910d74d4e2eb9ef49440c1fb6ad" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa289c674d34661bf49dd8bd7186a15" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2911b16ff0c74be48ae3a6eaa3db27cd" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:00:33.442996 19141 tablet_service.cc:1514] Processing DeleteTablet for tablet 500961d6706043ca9bd3fd2eb3f274a4 with delete_type TABLET_DATA_TOMBSTONED (TS 344bdf13092a447da55bd7a037da918a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41220
I20240503 14:00:33.443568 20920 tablet_replica.cc:330] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: stopping tablet replica
I20240503 14:00:33.443882 20920 raft_consensus.cc:2229] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:33.444077 20920 raft_consensus.cc:2258] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:33.445068 20920 ts_tablet_manager.cc:1875] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:33.447093 20920 ts_tablet_manager.cc:1888] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:00:33.447181 20920 log.cc:1196] T 500961d6706043ca9bd3fd2eb3f274a4 P 344bdf13092a447da55bd7a037da918a: Deleting WAL directory at /tmp/dist-test-taskmJWZN5/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714744724484359-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/500961d6706043ca9bd3fd2eb3f274a4
I20240503 14:00:33.447712 17998 catalog_manager.cc:4830] TS 344bdf13092a447da55bd7a037da918a (127.0.99.133:39929): tablet 500961d6706043ca9bd3fd2eb3f274a4 (table rebalance_test_table_0 [id=703e53a267474e24a924ca8e619241c9]) successfully deleted
I20240503 14:00:33.547304 19141 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:33.547295 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.556344 18265 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:33.559334 18397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.560295 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:33.647022 19141 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:33.647457 19010 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.649260 18265 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:33.655743 18135 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:33.664410 18397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.699512 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 18073
W20240503 14:00:33.706115 18221 connection.cc:535] server connection from 127.0.99.129:48609 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:00:33.706631 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 18204
I20240503 14:00:33.713184 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 18336
I20240503 14:00:33.720088 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 18948
I20240503 14:00:33.726861 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 19079
I20240503 14:00:33.733812 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmJWZN5/build/debug/bin/kudu with pid 17982
2024-05-03T14:00:33Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (20778 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (37945 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (109265 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>
1 FAILED TEST